This job was posted by https://illinoisjoblink.illinois.gov : For more information, please see: https://illinoisjoblink.illinois.gov/jobs/12579211 Senior Data Analyst

College Board - Learning & Assessments (L&A) Division

Location: This is a fully remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office).

Type: This is a full-time position

About the Team

The Data Management & Operational Support (DMOS) team serves as the gatekeeper and steward for all data-including content, metadata, and statistics-coming in and out of Assessment Design & Development (AD&D) to downstream stakeholders. We guide the department in adopting best data practices and partner with Test Security to maintain the integrity and security of all our test content-both within the organization and in the testing field. We ensure, where possible, the highest degree of quality of our data so the student testing experience is flawless and their score accuracy is upheld. We partner with Psychometrics, Applied Psychometrics Research, and Math and Reading & Writing subject matter experts to conduct intensive data analyses that inform evaluation of the efficacy and validity of our assessment instruments, and support the efforts of the State Work team to increase adoption of the digital SAT Suite at state and district levels. In addition, we are both key stakeholders and consultants on development of College Board\'s end-to-end digital assessment ecosystem, from our item development platforms to digital delivery, scoring, item calibration, and reporting.

About the Opportunity

As our Senior Data Analyst, you are recognized as a subject matter expert for all data-including content, metadata, and statistics-coming in and out of Assessment Design & Development. You work closely with colleagues in Content, Applied Psychometrics Research, Psychometrics, and Research to understand their business needs and are responsible for analyzing and interpreting large amounts of student SAT test data to guide data-driven strategy at College Board, inform ongoing assessment item development and pretesting, and improve the student test experience.You work on complex problems where analysis of situations or data requires an in-depth evaluation of multiple factors and frequently contributes to the development of new business strategy, ideas, and methods. You exercise significant independent judgment to determine the best method for accomplishing work and achieving objectives in a fast-paced and deadline-driven environment.

In this role, you will be responsible for:

Assessment Research Support (70%)

• Conduct comprehensive data analysis and develop tools to support innovative AI projects, including automated essay scoring (AES) and pretest item statistics prediction. These initiatives aim to significantly increase scoring efficiency and streamline the item development process, resulting in substantial cost reductions for human reading in SAT essays and AP free response questions and lightened sample size requirements for pretesting.
• Fulfill both ad hoc and planned data requests - from various stakeholders internally at AD&D, Psychometrics, Research, Data Science and/or downstream systems - related to SAT Item metadata, statistics and usage, and student test question timing data from all SAT administrations.
• Develop and implement strategies for making data available to AD&D Content staff, Applied Psychometric Research, Psychometrics, and others, following established processes and protocols to ensure data accuracy and consistency in reporting
• Collect, load, clean, profile, and analyze large amounts of student testing data
• Present findings and analysis to requestors and stakeholders across the organization, and lead any follow-up nalysis or reporting requests

Assessment Operational Support (30%)

• Conduct analysis and provide descriptive data-driven reports to stakeholders in support of SAT test assembly and pretest design; e.g., aggregate summaries, frequencies, rosters, cross-tabulations, correlations, and regression analyses
• Participate in item challenge investigations alongside DMOS team members, from triaging initial challenge submission through disposition and communication back to the challenge issuer (student)

About you, you have:

Ideally 10+ years of experience in supporting assessment content development workflow

Expertise in data exploration using SQL including ability to create complex queries

Proficiency in programming languages such as Python or R to integrate with machine learning (ML) algorithms and large language models (LLMs) and predictive modeling techniques to enhance research and development

Experience with relational data modeling and metadata management, leveraging relational databases and/or data warehouse to drive business reporting solutions

Excellent attention to detail and proven ability to break down and convey complex information to both technical and non-technical audiences effectively.

Must be resourceful, able to adapt to new si