Skip to main content

NCAS Manager of Assessment & Psychometrics


​Position: Regular, Full-Time
Location: 200 Granville St, Vancouver, BC. (NOTE: A combination of home-office (remote) and office-based work is expected.)​
Start date: December 2021
Application deadline: Applications will be accepted on a rolling basis until the position is filled

The opportunity

NCAS is looking for an experienced psychometrician to support its current and future assessments. This Vancouver-based position provides psychometric and data flow management expertise to the ongoing refinement and evolution of NCAS’ multi-pronged, multi-dimensional assessments that involve varied assessment tools such as Objective Structured Clinical Examinations (OSCE), oral interviews, and selected-response measures. 

NCAS is disrupting the assessment space with its innovative approach to assessment. This position offers an exciting opportunity for an exceptional individual interested in developing and implementing new competency assessment models, strengthening reliability and validity approaches, and applying foundational psychometric and statistical research to improve the quality of health care across Canada.

Who we are

The Nursing Community Assessment Service (NCAS) is part of the BC College of Nurses and Midwives. BCCNM is empowered under the Health Professions Act to regulate the practice of all licensed practical nurses, nurse practitioners, registered midwives, registered nurses, and registered psychiatric nurses in British Columbia. Regulation helps to protect the public by ensuring that professional care or service received by the public is competent, ethical, and meets the standards that society views as acceptable. NCAS operates as a unique and independent unit within BCCNM. 

BCCNM launched NCAS in January 2017 to assess the competencies of professionals seeking to enter nursing practice in BC and other parts of Canada. NCAS is a testing service that regulators rely upon to determine if someone is safe to enter practice. Our high-stakes assessment is comprised of a computer-based test that is administered at testing sites all over the world, as well as a simulation-lab and oral assessment that are delivered in Vancouver and Halifax. The NCAS assessment represents a consistent, rigorous, and defensible approach for determining the extent to which individuals possess the competencies required to enter practice safely. 

At BCCNM, we value and celebrate diversity. We are committed to fostering an open, welcoming, and inclusive work environment where we value and respect all perspectives. We aspire to employ staff who are representative of the diverse B.C. communities we serve. We hire based on merit and encourage qualified applicants of all backgrounds to apply.

Each of us strives to meet our commitment to the following values to maintain a workplace that is welcoming, respectful, and supports the well-being of every BCCNM employee:

  • Trust is essential
  • Better together​
  • Speak the truth
  • Protect the public
  • Stay curious ​​

What we offer you

Our office is located at 200 Granville Street in downtown Vancouver and on the unceded territories of the Musqueam, Squamish and Tsleil-Watuth nations. This position is ideal for an individual who is comfortable working in a hybrid remote and in person environment. 

As a member of the team, you can expect to be working in a collaborative, team-based environment and to be treated in a respectful and professional manner. The College is committed to employee professional development and career growth. We promise to provide you with rewarding work that challenges you. While your workload will be busy, the College supports staff in achieving a healthy work-life balance.​

What you'll be doing

Reporting to the Director of Assessment, Innovation, and Operations, the Manager of Assessments and Psychometrics divides their time between three principal accountabilities: providing psychometric and data management expertise to support the delivery of existing assessments; leading the development of new assessment tools; and, creating and/or evolving the dataflow pipelines to optimize data integrity and organization. The manager establishes the scoring methodologies and design of assessment instruments for the production of results and analysis of item and test functioning. The Manager provides oversight to assessment development staff while also playing a critical role in designing and ensuring the quality of data management pipelines.

Responsibilities​​​

  • In consultation with the Director of Assessment, Innovation, and Operations (Dir, AIO), manage the development, implementation and assessment of the tools and processes for the assessment of health care competencies.
  • Lead the development, refinement, and analysis of performance-based and selected-response instruments, including pilot and field testing, standard setting, and validation activities needed for the assessment of health care competencies.
  • Provide oversight to assessment development staff including an Exam Developer and Junior Psychometrician.
  • Create and/or refine, manage, and troubleshoot data flow pipelines (in R and SQL) that enable assessment scoring and reporting.
  • Generate performance reports, with associated quality checks (include importing data via Scantron, Excel, XML, and SQL in R)
  • Identify risks and offer recommendations as appropriate
  • Interface with multiple stakeholders and technology providers to ensure accurate and seamless data flows.
  • Collaborate with Director, AIO to understand the purpose of each new assessment in order to define required content and outcomes.
  • Create and implement quality-control procedures for assessment delivery, including rater scoring.
  • Develop, implement, and evaluate standard-setting and product-improvement processes, including the facilitation of discussions with subject matter experts and stakeholders.
  • Provide regular psychometric consulting, problem-solving and innovation in support of assessment operations and new assessment builds, including item analysis, forms construction and equating, reliability and validity analyses, etc.
  • Work closely with the Exam Developer to create and launch training for content writers and certification tools for raters, and to conduct analysis in support of content development.
  • Draft public and technical reports, as well as documentation on the validity and reliability of the assessments.
  • Identify and carrying out independent research in support of assessment activities.
  • Present research findings, representing the organization and sharing innovations at professional conferences and/or through publication of research articles
  • Foster and maintain an organizational culture that promotes mutual respect, teamwork, and service excellence.

Education & skills​

  • Minimum Masters (Ph.D. preferred) in Measurement and Evaluation, Data Science, Psychology, Statistics, or educational equivalent in a closely related field.
  • At least five years of experience in an applied measurement or data science environment.
  • At least five years in designing and conducting statistical and/or psychometric analyses.
  • Expertise in a variety of psychometric models (e.g., classical test theory, generalizability theory, factor analysis, item response theory, differential item functioning, dimensionality analysis, interrater reliability analysis).
  • Expertise in assessment, statistical, and database software (RStudio and R Markdown (required), and other programming languages an asset; Microsoft SQL Server Management Studio, XML file processing, SQLite, Remark OMR, Qualtrics, Excel) and the ability to write user-defined R functions.
  • Familiarity with professional assessment standards (e.g., INACSL, AERA/APA/NCME Standards)
  • Experience managing testing programs with small volumes is preferred
  • Experience managing staff is preferred
  • Strong project management background, preferably in technical education, certification, or credentialing services. Aptitude to quickly come up to speed on new concepts.
  • Exceptional research and writing skills.
  • Demonstrated initiative-taker, with strong organizational skills and the ability to meet deadlines.
  • Demonstrated ability to prepare analytic plans, monitor progress, and track timelines.
  • Highly collaborative, with strong interpersonal skills to work with various stakeholders in the assessment development and reporting process.
  • Ability to work independently and collaboratively within a team.
  • Demonstrated ability to prepare effective visual presentations of complex data.
  • Bilingualism in French and English is an asset
  • Ability to thrive and take initiative in an environment of ambiguity and change.
  • Displays a passion for doing meaningful work
  • Utilizes available networks and resources to stay current in the field of competency assessment
  • Demonstrated understanding of diversity, inclusion, and cultural humility as they apply to nursing and midwifery practice and health care is an asset.​

How to apply

We offer accommodation for applicants with disabilities to take part in the selection process. If we contact you about an interview or testing, please let us know if you require accommodation. We keep all information in relation to accommodation confidential. 

Please forward your resume and cover letter to careers@bccnm.ca. Please use Manager of Assessment & Psychometrics in the subject line of your email. We will accept applications until the position is filled. 

Note: References, education and professional credential verifications, and a criminal record check will be required for all final candidates. 

While we appreciate all responses, only shortlisted applicants will be contacted. 

To learn more about NCAS, please visit www.ncasbc.ca. Thank you for your interest in NCAS and the BC College of Nurses and Midwives.​​​​​​​​​​​​​​​​​