From an English label to a practical decision Definitions and primary sources. Third-party raw scores and prices are not reproduced here.
Decision Choice correctness, confidence, and the cost of error. Evaluate separately from speed and explanation quality.
Openness Weights availability, licensing, and transparency. Hardware requirements and deployment feasibility require separate verification; openness is not an intelligence score.
Term in English or Russian FamilyAll families Decision — decision quality Openness — openness and transparency Agents and task execution Data and extraction quality Research and reproducibility Cost and resources Economics and efficiency Probability and confidence Our project metrics
Matching terms: 88
Decision — decision quality What it measures A weighted assessment of structured decisions, confidence and invariance. Why it matters A research lead for routing, classification and extraction. Freeze the component version. Limitation Five Decision Scores plus 15% Symmetry. It is not a project success probability or the Jev scale. What it measures For a matching evaluation: 50 × (1 + accuracy − mean((p − c)²)); c is chosen-answer correctness. Why it matters Penalizes confident mistakes; accuracy is a fraction from 0 to 1. Limitation This component formula does not reconstruct the full v2 index. Ground truth is required. What it measures Separate public reproduction and private Full evaluation; Public adjusts for chance. Why it matters Use the public kit to test exact engines with pinned data and protocol. Limitation Full uses 60% private same-skills and 40% private new-domain tests. Public is not a reproduction of Full. Openness — openness and transparency What it measures Access to weights, licenses and transparency of data, methodology and training code. Why it matters Helps assess self-hosting, reproducibility and supplier dependence. Limitation V1.0 normalizes raw points by 18. Openness does not establish accuracy, hardware fit or commercial permission by itself. Agents and task execution What it measures Average objective completion per task; a task with a guardrail breach receives zero. Why it matters Relevant to multi-application AIOS workflows and controlled document processing. Limitation Partial objectives are not fully accepted tasks. Simulated apps do not certify your production tools. Agents and task execution / % objectives Objectives Completed Выполненные цели
What it measures Completed objectives, including tasks with violations. Why it matters Read together with acceptance and violation metrics. Limitation The difference from Score is not itself a violation rate; aggregation can differ. What it measures Tasks with every objective met and no guardrail violations. Why it matters Closer to acceptance of a complete business result. Limitation Do not substitute an objective completion percentage. What it measures Composite software-engineering evaluation of an agent with its model and execution settings. Why it matters Compare the full model–harness pair and its measured resource budget. Limitation Versioned components matter. API-equivalent task cost is not a subscription invoice or business-success cost. What it measures Coding exercise results under Aider-specific editing and test conditions. Why it matters Useful when selecting a model and edit format for an Aider workflow. Limitation Do not transfer its rank to another executor, repository or retry policy. What it measures Repository issue resolution judged with a specified evaluation harness. Why it matters A useful engineering probe before testing patches in your own repositories. Limitation Pin dataset, agent, model, tools and attempt budget. A pass is not production deployment readiness. Agents and task execution / % tasks Terminal-Bench Задачи в терминале
What it measures Completion of versioned tasks in a terminal environment. Why it matters A probe for tools, environment handling and task completion. Limitation Versions, timeouts, compute and harnesses change the meaning of the score. What it measures Adherence to specified instructions under the benchmark evaluator. Why it matters Helps design tests for formats, constraints and structured outputs. Limitation Instruction compliance does not establish factual correctness or authorization. What it measures The exact model, provider, effort, harness, tools, memory and permissions used for one run. Why it matters Makes a result reproducible and attributable. Limitation A short alias or a provider logo is not a complete identity. What it measures Software that manages the agent loop, tool interfaces, context and evaluation. Why it matters The same model may behave differently under another harness. Limitation DSH Harness and DeepSeek Harness are distinct identities. What it measures Model interaction steps and individual tool invocations; record them separately. Why it matters Explain time, orchestration overhead and recovery behavior. Limitation Fewer calls may mean efficient work or premature failure; inspect accepted outcomes. What it measures The tools, resources, data and actions explicitly available to a run. Why it matters Defines what autonomy can be tested and promoted. Limitation A benchmark score does not authorize tender submission, live trading or production deployment. Data and extraction quality What it measures Quantitative questions answered using spreadsheet and document workflows. Why it matters A probe for analytical assistants in all three projects. Limitation Check source lineage, units, numerical oracles and reproducibility locally. What it measures Correct answers grounded in long supplied material. Why it matters Useful for tender packs, research logs and specifications. Limitation Maximum context capacity does not establish reliable retrieval at every position. What it measures Evaluate on untouched future periods with only information available at the decision time. Why it matters Tests whether a result survives outside its tuning period. Limitation Repeatedly selecting on the same holdout turns it into development data. Research and reproducibility What it measures A versioned aggregate across several capability evaluations. Why it matters Use individual components to build a shortlist for a task. Limitation A broad ranking is not a calibrated probability of project success. What it measures Colors rescaled within the selected comparison group. Why it matters Helps inspect patterns across task groups. Limitation Green is not an absolute success probability; changing selected models changes colors. What it measures A factorial comparison separates model, executor and their interaction on matched tasks. Why it matters Reveals which pairing works best for a particular task family. Limitation A strong model does not guarantee the best pair; sparse cells cannot support reliable interaction estimates. What it measures A frozen baseline compared with one versioned candidate under declared gates. Why it matters Turns automatic generation into a controlled improvement process. Limitation Keep a protected holdout, all failed trials, a budget, a stopping rule and rollback evidence. Research and reproducibility Accuracy Доля правильных ответов
What it measures correct answers / all scored questions Why it matters Read with the same taskset, unit and budget. Limitation Class imbalance can hide missed rare events. What it measures true positive findings / all positive findings Why it matters Read with the same taskset, unit and budget. Limitation Precision alone does not show what was missed. Research and reproducibility Recall Полнота обнаружения
What it measures found true positives / all true positives Why it matters Read with the same taskset, unit and budget. Limitation Recall alone does not show false alarms. What it measures 2 × precision × recall / (precision + recall) Why it matters Read with the same taskset, unit and budget. Limitation State the macro/micro weighting and zero-denominator convention. Research and reproducibility pass@1 Успех первой попытки
What it measures success under a single declared attempt Why it matters Read with the same taskset, unit and budget. Limitation Pin the evaluator and the attempt budget. What it measures at least one accepted solution among k attempts Why it matters Read with the same taskset, unit and budget. Limitation A higher k changes cost and does not demonstrate repeat reliability. Research and reproducibility pass^k / all-K Успех во всех k повторах
What it measures all of the fixed k repeats accepted for a task Why it matters Read with the same taskset, unit and budget. Limitation Use the task as the cluster; do not assume p^k without independence. What it measures response-start delay / generation throughput Why it matters Read with the same taskset, unit and budget. Limitation Fast output is not fast accepted completion; queues, tools and retries remain. Research and reproducibility Percentage points Процентные пункты
What it measures absolute difference between two percentages Why it matters Read with the same taskset, unit and budget. Limitation 80% to 85% is +5 points and +6.25% relative, not the same change. Research and reproducibility Log scale Логарифмическая шкала
What it measures equal distances correspond to equal ratios Why it matters Read with the same taskset, unit and budget. Limitation Zero and negative values are not defined on an ordinary logarithmic axis. Cost and resources Cost and resources Cost per session Стоимость сессии
What it measures The cost of a session under a stated length, pricing and cache condition. Why it matters Useful for cost planning when the session mix is known. Limitation A 1-turn session is not comparable with a 50-turn task; no quality is implied. What it measures A token-based estimate at a stated provider tariff. Why it matters Supports reproducible price scenarios. Limitation Keep estimate, billed API amount, subscription allocation and full cost separate. What it measures Distinct billing and usage categories whose definitions depend on the provider. Why it matters Count retries, assistants and auxiliary models in the same cohort. Limitation Do not count reasoning twice when it is already included in output tokens. Economics and efficiency Economics and efficiency Pareto frontier Граница Парето
What it measures Candidates not dominated on every chosen objective within a comparable experiment. Why it matters Shows quality–cost–latency trade-offs before choosing business priorities. Limitation Missing costs, different tasksets and statistical uncertainty can invalidate a claimed frontier. What it measures The value and publication date that were actually known at a past decision time. Why it matters Keeps macro scenarios and FinEdg research free of revised-data leakage. Limitation The value displayed today may differ from what the market knew then. What it measures Revenue minus variable cost differs from the timing of actual receipts and payments. Why it matters Supports viable pricing, bid/no-bid choices and budget stress scenarios. Limitation Business margin, trading PnL and benchmark quality have different units and must remain separate. Probability and confidence Probability and confidence Wilson 95% interval 95%-й интервал Уилсона
What it measures An interval for a Bernoulli success proportion under its sampling assumptions. Why it matters Shows the uncertainty hidden by a single observed percentage. Limitation Repeated calls on one task are not independent observations; this is not a Bayesian posterior probability. Probability and confidence Beta–binomial model Бета-биномиальная модель
What it measures Beta(a,b) prior plus k successes in n independent trials gives Beta(a+k,b+n-k). Why it matters Makes prior beliefs and their update explicit. Limitation State and test the prior; dependence and changing task difficulty require a richer model. Probability and confidence Expected loss Ожидаемые потери решения
What it measures Sum of outcome probabilities times their losses, including action and review costs. Why it matters Connects a calibrated prediction to a project-specific action. Limitation Loss assumptions must be explicit; a confidence threshold never creates permission. What it measures Compare systems on the same cases; resample independent task, company or time clusters. Why it matters Reduces task-mix noise and respects repeated observations. Limitation Separate comparisons across many candidates need a declared selection and multiplicity policy. Probability and confidence / squared_probability Binary event Brier Brier бинарного события
What it measures mean_i (p_i - y_i)^2; y in {0,1} Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures mean_i (p_correct_i - c_i)^2; c=1 iff the chosen answer is correct Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures mean_i sum_k (p_ik - 1[y_i=k])^2; sum_k p_ik=1 Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Probability and confidence / bin_probabilities Reliability diagram Калибровочная диаграмма
What it measures For frozen bins: count, mean forecast, observed outcome frequency and uncertainty Why it matters Interpret jointly with the other metrics; no universal direction. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures One-sided exact binomial upper bound on independent representative episodes; zero errors: 1-alpha^(1/n) Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics Our project metrics / fraction Accepted task rate Доля принятых задач
What it measures accepted unique task episodes / assigned eligible task episodes Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures accepted episodes with no human intervention / assigned eligible episodes Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction All-K repeat reliability Успех во всех K повторах
What it measures mean_i product_r accepted[i,r], fixed K Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / count Critical violation count Критичные нарушения
What it measures count of predefined forbidden state transitions / critical invariant failures Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures fault cases restored with same intended state and exactly-once business effect / all injected fault cases Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures required event links present / required event links Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures cases passing all unit-specific tolerances / all cases Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Feasible solution rate Соблюдение ограничений
What it measures solutions satisfying every declared constraint / all fixtures Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures For minimization: (candidate objective - certified optimum)/abs(certified optimum) Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures correct critical field values / all required critical field values Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Macro field F1 Средний F1 типов полей
What it measures mean over declared field types of 2*precision*recall/(precision+recall) Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Entity-link precision Точность связей компаний
What it measures correct proposed links / all proposed links Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Entity-link recall Полнота связей компаний
What it measures recovered true links / all true links in adjudicated set Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures claims semantically supported by cited versioned source / all asserted factual claims Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures correctly reported required facts / all required facts in gold Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Data-lineage coverage Происхождение данных
What it measures required fields with source version/hash/location/as-of / all required fields Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / count Data-leakage count Случаи утечки данных
What it measures violations of train/calibration/test or as-of boundaries Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures critical outputs using superseded or not-yet-available source versions Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures eligible assigned accounts with matured qualified outcome / eligible assigned accounts Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures net cohort revenue minus variable acquisition/service/refund costs, divided by all eligible assigned accounts Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures goodness-of-fit p-value for observed vs planned assignment counts Why it matters Interpret jointly with the other metrics; no universal direction. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures material promises without evidence / prohibited claim instances Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures all relevant costs of cohort / accepted unique task episodes Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Required cost-item coverage Полнота учёта расходов
What it measures cost components backed by receipts or explicit allocations / required components Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures max absolute residual after prescribed currency-specific rounding Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures net revenue excluding chosen indirect-tax treatment - variable cost Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures min_t(initial cash + cumulative dated receipts - cumulative dated payments) Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures min over declared stress scenarios of contribution Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Experiment manifest coverage Полнота манифестов
What it measures episodes with complete frozen config/data/criteria/cost lineage / episodes Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / minutes/accepted_task Human review minutes per accepted task Время участия человека
What it measures all logged intervention and review minutes / accepted unique tasks Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures p95 wall-clock elapsed time including queue/retries/review; report timeout mass separately Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / count Missed critical-deadline count Пропущенные дедлайны
What it measures missed gold critical deadlines with explicit timezone/version Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures finished/expired episodes lacking correlated terminal result Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures unauthorized development access to protected inputs/labels before final evaluation Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures artifacts independently rerun with equivalent outputs within frozen tolerances / required reruns Why it matters Higher, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / count Material regression count Критичные регрессии
What it measures new failures on predeclared critical regression cases Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures episodes exceeding their explicit monetary/resource/action limits Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. What it measures equity_end/equity_start - 1, after recorded fees/funding/slippage under declared simulation Why it matters Interpret jointly with the other metrics; no universal direction. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / fraction Maximum drawdown Максимальная просадка
What it measures max_t(1 - equity_t / running_peak_equity_t) Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment. Our project metrics / count PAPER execution invariant failures Ошибки PAPER-исполнения
What it measures duplicate orders, stale prices, invalid sizing or violated approved controls in PAPER tests Why it matters Lower, subject to the acceptance gates. Limitation A proposed local metric. Freeze its denominator, scope and missing-data policy before running the experiment.