AP Statistics Topic 1.12: Potential Problems with Sampling

01 · Start here

The concept in 60 seconds

Bias is a systematic tendency for an estimate to miss its target in a particular direction. A procedure might tend to make an average too low or a proportion too high. The issue is how the data are obtained, not simply how many observations there are.

Ask three practical questions: Could everyone of interest be selected? Who actually supplied data? Are the recorded answers accurate? Those questions help locate a problem before you calculate a mean or proportion.

Follow the survey from the population to the reported result

1PopulationWho do we want to learn about?

2FrameWho is available for selection?

3Selected sampleWho is chosen, and by what rule?

4RespondentsWho actually provides data?

5Recorded answersWhat do the responses really measure?

A good selection rule is one step. Missing people, missing responses or distorted measurements can still affect the result.

A larger sample does not repair a flawed process. Asking more people from an incomplete frame does not bring the missing group onto that frame. Collecting more answers to a leading question does not make the wording neutral.

A potential source of bias does not tell you its exact size. You may be able to explain a likely direction, or you may need more information. A careful answer identifies the mechanism and avoids inventing what unobserved people would have said.

02 · Inspect the whole process

Cedar High reviews its travel-time survey

Cedar High wants to estimate the mean one-way home-to-school travel time, in minutes, for all $1{,}200$ enrolled students on a specified Tuesday. Students travel in different ways and live at different distances. Each proposal below targets that same population and measurement.

Four fictional proposals. Find the weak point in each collection process.
Proposal	What happens?	The concern to investigate
Use the parking-permit list	Select $120$ students randomly from permit holders, then describe the result as a school-wide estimate.	Students without permits cannot be selected from this frame. Their travel times may differ.
Post an open survey link	Any enrolled student may click a link headed “Tell us about frustrating journeys.” Only students who opt in supply data.	Interest in the topic may affect who volunteers. This is not a random sample of enrolled students.
Stop after $1\text{ invitation}$	Take an SRS of $120$ from a complete enrollment list. Only $42$ answer the first invitation; the other $78$ are ignored.	The responding students may differ from the selected students who did not answer.
Push a short-time answer	Ask selected students, “Your journey usually takes under $20 \,\mathrm{min}$ , doesn’t it?” and use the answers to describe Tuesday travel.	The question suggests a preferred answer and changes a specific Tuesday measurement into “usually.”

Randomizing the selection from a parking list does not solve its missing coverage. Randomizing from the full list does not ensure everyone responds. Even with full response, misleading wording can distort the recorded data.

Think first: Which proposal starts with a good random selection process but loses some of the selected students before obtaining data?

Check your prediction

The third proposal. The original $120$ are an SRS from the enrollment list, but data come from only $42$ respondents. The main concern is possible nonresponse bias, especially if the $78$ nonrespondents have systematically different Tuesday travel times. Their silence is not a $\text{travel time of }0$ .

03 · Separate two kinds of uncertainty

Bias versus chance variation

Two well-conducted random samples can produce different means even when neither method systematically favors high or low values. That sample-to-sample movement is chance variation. Bias describes a repeated tendency of a procedure, rather than the accidental difference in one sample.

Different questions about an estimate

Chance variationDoes the estimate move around?

Below target ↔ Above target

Different random selections give different results. In a suitable unbiased procedure, there is no persistent shift away from the target.

Systematic biasIs the process tilted?

Process → tendency too high or too low

A missing group or a distorted answer can repeatedly pull the estimate away from the population value.

This is a conceptual comparison, not a plot of simulated results. A biased procedure can still produce an occasional estimate near the true value.

The parameter is the population value of interest, such as the true mean Tuesday travel time. A statistic is calculated from the collected data, such as the respondents’ mean. You usually do not know the parameter, so you cannot measure the exact error just by inspecting the statistic.

One estimate is too highThis alone does not show that the sampling method is biased. A random sample can be above the target by chance.

The method omits a relevant groupThis supplies a reason to suspect bias, even before the true population mean is known.

Increasing the size of a properly collected random sample generally reduces random sampling variability. It does not automatically remove a systematic problem. A large online poll may estimate the opinions of its volunteers very precisely while giving a poor estimate for the entire population.

The vocabulary here concerns statistical bias. A collector does not have to intend to mislead: a convenient choice, confusing wording or an outdated list can create a problem accidentally.

04 · Locate the source

Recognize the main sources of bias

Undercoverage: part of the population is poorly represented

Undercoverage is a concern when a selection process leaves out part of the target population or makes a relevant part less likely to be selected. A list of parking-permit holders is an incomplete frame for all enrolled students. Bus riders, walkers and other students without permits cannot enter a sample selected only from that list.

Explain why the missing group matters for the variable being estimated. If students without permits tend to have longer travel times, a permit-only estimate may be too low. If you know nothing about their times, identify the coverage problem without claiming a direction.

Improve the process: use a current enrollment frame covering all students and an appropriate random selection rule. Increasing the number chosen from the permit list leaves the same group excluded.

Voluntary response: people choose themselves into the sample

An open invitation followed by a sample made entirely of people who opt in creates a voluntary response sample. People with a strong interest, strong feelings or more time may be more likely to take part. A journey-complaints poll may especially attract students unhappy with their travel.

The possibility of bias comes from self-selection and its connection to the response. Volunteers are not automatically dishonest. $100\text{ honest answers}$ from a particular kind of volunteer can still give an unbalanced picture of a much larger group.

Improve the process: select units using chance from the target population’s frame, then contact those units and use a thoughtful follow-up plan. Do not treat the first people willing to answer as a substitute for that selection process.

Nonresponse: selected units fail to provide data

Nonresponse occurs when some chosen units do not provide the requested data. They may not be reached, may decline, or may leave a relevant item blank. Nonresponse bias is a concern when respondents and nonrespondents differ in a way related to the study’s variable.

For Cedar High, suppose students with long journeys are especially unlikely to answer because the survey arrives during their commute. Their absence could pull the respondents’ mean below the full population mean. A low response rate raises concern, but the rate alone does not prove the direction or amount of bias.

Improve the process: follow up with the originally selected students, offer suitable response times or modes, and make the survey easy to complete. Replacing unavailable students with nearby volunteers changes the responding group and does not automatically fix the design.

Response bias: the supplied values are systematically distorted

A person can be selected and respond, yet supply an answer that differs from the true value. Leading questions, unclear wording, memory problems or pressure to give a socially acceptable answer can create a systematic shift. Measurement procedures can also be at fault: a scale that reads every object’s mass $2 \,\mathrm{g}$ too high creates a consistent upward error.

Ask whether the answers tend to be pushed in a particular direction. Ordinary mistakes can occur in either direction; not every isolated error establishes response bias. In a survey about arriving late, a student may underreport delays if answering directly to the staff member who records lateness.

Improve the process: define the measurement clearly, use neutral questions, choose a suitable recall period, provide a response setting that reduces pressure, and check instruments. Do not claim that anonymity or any single change guarantees perfectly accurate responses.

Similar-looking problems happen at different stages

UndercoveragePopulation → incomplete frame

Some relevant students cannot enter the selection pool.

Ask: Were they available for selection?

Voluntary responseOpen invitation → self-selected volunteers

People choose themselves into the data set.

Ask: Who decided which people would participate?

NonresponseChosen units → some missing answers

Some selected students do not supply the data.

Ask: Were they chosen but did not answer?

Response biasRespondent → distorted recorded value

An answer is present, but may be systematically inaccurate.

Ask: Did the response measure the truth?

Convenience sampling: choose whoever is easy to reach

Asking the first $60$ students near $1$ gate is a convenience sample. Ease of access determines inclusion. Students using another entrance, arriving at a different time or absent that morning may be missed. If arrival patterns are related to travel time, the result can be systematically unrepresentative.

Convenience and voluntary response are both nonrandom selection methods, but the mechanisms differ: the collector chooses accessible units in the first; participants opt into an open invitation in the second. A study can contain both mechanisms, along with other problems.

Giving consent is not the same as creating a voluntary response sample. In an SRS survey, people are selected first and remain free to decline. Missing answers then create a nonresponse concern. In an open poll, people choose themselves into the sample in the first place.

Not every unequal selection chance signals a bad design. Topic 1.11 allowed different sampling fractions in strata, with suitable analysis. Here the concern is a selection or collection process that systematically misrepresents the target, rather than any planned unequal probability by itself.

05 · Collect the intended information

Improve questions and measurement

A question should make the intended variable clear without suggesting the answer the researcher wants. For a travel-time measurement, specify the journey, day, starting point, ending point and units. “Tuesday” and “a typical week” are different targets.

From a pushed answer to a defined measurement

Problematic wording

“Your journey usually takes under $20 \,\mathrm{min}$ , doesn’t it?”

Suggests a short journey, asks for a yes/no judgment, and uses an undefined “usually.” It does not collect the specific Tuesday duration.

Improved wording

“On Tuesday 6 October, how many minutes passed from leaving home to arriving at school for your morning journey?”

Names the day, journey and units without proposing a preferred duration. In a real survey, replace the example date with the actual study date.

These are original teaching examples. Clear wording helps; recall and recording errors may still occur.

Small wording choices can change what an answer represents.
Problem	Example	A better approach
Loaded description	“Do you support the wasteful new bus route?”	Describe the route neutrally, then ask for support, opposition or no opinion.
Two issues in one item	“Are the buses punctual and comfortable?”	Use separate punctuality and comfort questions. A respondent could answer differently to each.
Unclear reference period	“How often are you late?”	Specify a period and define what counts as late, such as arriving after the scheduled start on each school day last week.
Unbalanced response choices	Only “good,” “very good” and “excellent” are offered for a service rating.	Allow ratings below and above neutral, with applicable options such as “did not use the service.”

Self-reported data are not automatically useless. Ask whether people can recall the information and whether the setting encourages accurate reporting. Recording a journey’s start and finish when it happens may reduce recall problems compared with guessing its duration weeks later.

A neutral question does not restore missing population coverage. A complete frame does not calibrate a faulty instrument. Match the improvement to the stage where the error occurs, and keep that distinction in your explanation.

06 · Follow the reasoning

Four worked examples

All scenarios and quantities below are invented for teaching. The reasoning shows how a collection process could affect an estimate; it does not report findings from real surveys.

Example 1: Undercoverage can shift a mean

Task: a fictional school has $900$ nearby students with a population mean Tuesday travel time of $20 \,\mathrm{min}$ and $300$ farther-away students with a population mean of $40 \,\mathrm{min}$ . A researcher randomly samples only from the nearby-student list, but wants to estimate the mean for all $1{,}200$ students.

Name the target: the mean travel time of all $1{,}200$ enrolled students.
Locate the missing group: the $300$ farther-away students are absent from the frame, so they cannot be selected.
Use the supplied group information: farther-away students have the higher mean. Their exclusion removes a group that should pull the school-wide mean upward.
State the possible effect: a nearby-only estimate tends to underestimate the full school’s mean.

\text{Whole-school mean}=\frac{900(20)+300(40)}{1{,}200}=25\,\mathrm{min}

$\text{Covered-group mean}=20\,\mathrm{min}$ ; $\text{difference from the target}=20-25=-5\,\mathrm{min}$ .

The group means are known only because this is a constructed example. Random sample means from the nearby group may vary around $20 \,\mathrm{min}$ , but collecting more nearby students does not include the farther-away group. The $5$ -minute gap shows the coverage problem’s direction in this illustration; it is not a formula for calculating unknown bias in an ordinary survey.

Improve: select randomly from the complete enrollment list, or use an appropriate design covering both groups. If stratifying, use the population shares correctly in the whole-school estimate.

Example 2: Nonresponse is about chosen people

Task: a library selects an SRS of $200$ registered members to estimate mean books borrowed during September. $80\text{ members}$ respond. The study description states that frequent borrowers are more likely to respond than infrequent borrowers. Identify the concern and its likely direction.

Separate selection and response: $200$ members were selected; only $80$ supplied data.
Name the source: possible nonresponse bias. The concern arises after the random selection, because selected members do not all answer.
Connect the missing answers to the variable: the responding group disproportionately contains frequent borrowers.
State the direction: the respondents’ mean number borrowed may overestimate the mean for all registered members.

The response fraction is $\frac{80}{200}=40\%$ , and $120$ selected members did not respond. These counts alone would not establish an upward bias. The stated difference in borrowing behavior supports that direction.

Improve: follow up with the originally selected members using suitable contact methods. Where appropriate and permitted, a consistent borrowing-record measurement might reduce reliance on remembered counts. Do not record unanswered surveys as $0\text{ books borrowed}$ .

Example 3: Voluntary responses do not represent everyone

Task: a recreation center posts “Unhappy with our booking system? Tell us!” on its website. Of the $500$ visitors who submit the optional poll, $70\%$ report dissatisfaction. The manager concludes that $70\%$ of all registered members are dissatisfied.

Name the target population: all registered members, not just website visitors who answered.
Identify how inclusion happened: people chose themselves into an open poll. This creates a voluntary response sample.
Use the invitation’s focus: it specifically attracts people unhappy with booking. Dissatisfied members may be more motivated to participate.
Assess the conclusion: $70\%$ describes the poll’s respondents. Applying it to all members may overestimate the dissatisfied proportion.

There may also be undercoverage if some registered members do not use the website. The leading invitation adds another concern. When several problems occur together, identify the requested source and support it with the actual selection or wording detail.

Improve: take a random sample from a complete member frame, contact the selected members, use a neutral invitation and question, and follow up thoughtfully. $5{,}000\text{ self-selected responses}$ to the same poll would still leave its selection problem.

Example 4: An answer can be present but inaccurate

Task: students are randomly selected from a complete list. Every selected student answers a survey about days late to school last week, face-to-face with the staff member who issues lateness notices. The prompt states that students tend to report fewer late days to avoid embarrassment.

Check for missing people: the frame is complete and all selected students respond; the stated problem is not nonresponse.
Check the recorded values: students systematically underreport their actual number of late days.
Name the source: response bias, associated here with pressure to give a socially acceptable answer.
State the direction: the mean reported number of late days may underestimate the true mean for the target students.

Improve: define the period clearly and use a suitable confidential collection setting that reduces this pressure. A consistently defined attendance-record measure may be useful if appropriate for the study. Explain why the change addresses reporting, while recognizing that no collection mode guarantees $0\text{ error}$ .

07 · Show the mechanism

Explain a possible effect in context

A label such as “undercoverage” is only the beginning. A complete explanation connects the collection detail to the relevant difference and then to the estimate.

Build the explanation one link at a time

Collection detailOnly nearby students are on the frame

Farther-away students cannot be chosen.

Relevant differenceFarther-away students have longer journeys

This relationship is supplied in the example.

Possible effectSchool-wide mean may be underestimated

The excluded higher values would raise the target mean.

This is the undercoverage example’s reasoning, not a rule that all undercoverage lowers estimates.

Writing frame: “This procedure has a potential source of [type] bias because [specific selection, response or measurement detail]. [Excluded, overrepresented or misreporting group] may have [relevant difference]. As a result, [named statistic] may [overestimate / underestimate] [named population parameter].”

Adapt the frame to the evidence. Say “may” or “could” when assessing potential bias. If the description supplies a clear tendency, explain it; if it does not, say the direction cannot be determined from the information given.

Make the target and direction explicit.
Incomplete answer	Stronger answer
“It is biased because some people are missing.”	“The parking-permit frame excludes students without permits. If their journeys are longer, the permit-only mean may underestimate the mean Tuesday travel time of all enrolled students.”
“Nonresponse makes the answer too low.”	“In the library example, frequent borrowers are more likely to answer, so the responding members’ mean may overestimate the mean number of September books borrowed by all registered members.”
“The open poll is wrong.”	“The $70\%$ is the dissatisfied proportion among voluntary poll respondents. Their self-selection means it should not automatically be used as the dissatisfied proportion among all registered members.”

When the direction is unknown

A researcher’s list excludes members who joined this month, but the prompt says nothing about their opinions. You can identify undercoverage: those members cannot be chosen. You cannot conclude whether the satisfaction proportion is too high or too low without information relating membership age to satisfaction.

When several sources are possible

An online survey can have limited coverage, self-selection and a leading question at the same time. The delivery mode alone does not identify the method: an online questionnaire sent to an SRS can be a valid collection option, while a public link creates a different selection process. Explain the action described, rather than treating “online” as a diagnosis.

08 · Find the wrong turn

Common mistakes and how to fix them

Use the study’s mechanism as evidence.
Mistake	Why it fails	Better approach
“A large sample cannot be biased.”	Size does not repair missing coverage, self-selection or distorted responses.	Inspect the collection process before relying on the count.
Calling every missing person nonresponse.	A person absent from the frame was not selected and then lost.	Distinguish availability for selection from failure to answer after selection.
Calling a refusal response bias.	A refusal supplies no requested answer; response bias concerns distortion in supplied values.	Check whether the data are absent or systematically inaccurate.
“Everyone agreed to answer, so it is voluntary response sampling.”	Consent does not describe how people entered the sample.	Find whether selection preceded consent or people opted into an open invitation.
“Every kind of bias lowers the result.”	Different mechanisms can push estimates upward or downward.	Connect the group difference or reporting tendency to the specific mean or proportion.
“Only $40\%$ replied, so upward bias is proven.”	The response rate does not tell you the nonrespondents’ values.	Explain how response behavior is related to the study variable, if that information is available.
“The sample mean differs from the population mean, so the method is biased.”	One difference can result from chance variation.	Look for a systematic feature of the procedure, not merely one inaccurate estimate.
“Random selection or a census eliminates all error.”	Missing responses and distorted measurements can remain even after good selection, or when everyone is targeted.	Inspect coverage, actual response and measurement as separate steps.

Bias describes a systematic tendency, not just one difference.

Solution and explanation

No. This estimate is $2 \,\mathrm{min}$ above the parameter, but that difference can occur by chance in random sampling. It does not establish that the method consistently overestimates the mean. Inspect the selection and measurement procedure for systematic problems. A complete response count does not, by itself, verify measurement accuracy.

10 · Keep the essentials

Quick revision checklist

BiasA systematic tendency to miss the target in one direction.

Chance variationEstimates move between random samples; one miss does not prove bias.

UndercoverageA relevant part of the population is absent or poorly included in the selection process.

Voluntary responsePeople choose themselves into an open invitation’s sample.

NonresponseSome selected units do not provide data; relevant differences may distort results.

Response biasSupplied answers or measurements are systematically inaccurate.

Convenience samplingChoose accessible units rather than using a population-based chance process.

DirectionUse the relevant group difference; admit when it is unknown.

Quick questions students often ask

Is every online survey a voluntary response sample?

No. “Online” describes a collection mode. A questionnaire sent to randomly selected people is different from a public poll anyone can choose to enter. Examine the selection process and actual response, not just the device used.

Does a low response rate prove bias?

No. It increases concern about missing data, but a rate does not describe the missing values. Nonresponse bias depends on relevant differences between respondents and nonrespondents and how those differences affect the estimate.

Can a census have response problems?

Yes. A complete census obtains relevant data from every member of the population, so there is no variability from choosing a sample. A census attempt can still miss units or receive unanswered items; even complete coverage does not remove distorted measurements. Attempting to ask everyone does not guarantee accurate data from everyone.

Does bias always mean deliberate dishonesty?

No. A researcher may use an outdated list, people may misunderstand an item, or an instrument may be miscalibrated without anyone intending to mislead. Explain the statistical mechanism rather than assigning motives.

11 · Put the ideas together

Final understanding check

A fictional school wants mean Tuesday travel time for all $1{,}200$ students. It uses a current complete enrollment list and selects an SRS of $120$ students. $48\text{ answer}$ $1\text{ invitation}$ asking, “Your journey is normally short, under $20 \,\mathrm{min}$ , right?” The prompt states that students with longer journeys are less likely to answer and that the wording encourages people to describe their journeys as short.

Identify the selected sample size and responding count.
Explain two distinct sources of possible bias using the given details.
State a likely direction for the reported picture of journey length.
Explain why taking another $120$ students with the same collection process does not automatically solve the problem.
Suggest a targeted improvement for each of the two concerns.

Reveal the full solution

1. Counts: $120$ students are selected; $48$ respond. The simple response fraction is $\frac{48}{120}=40\%$ ; $72$ selected students do not answer. Do not confuse the planned sample with the responding group.

2. Nonresponse: selected students with longer journeys are less likely to answer. Their missing data can make the responding group overrepresent shorter journeys.

Response bias and measurement mismatch: the wording suggests a short-time answer and asks about “normally,” rather than the specified Tuesday. It also collects a threshold judgment instead of the numeric duration needed to calculate a mean. The resulting answers do not directly supply the study’s intended variable.

3. Direction: given the stated response pattern and pressure toward “short,” the reported picture may make journeys look shorter than they are in the full population. However, this yes/no item cannot directly produce a mean travel time; even a reported mean would need actual duration measurements.

4. Size: choosing more students while retaining the same $1$ -invitation and leading-question process leaves those mechanisms in place. More answers alone do not restore missing long journeys or collect the intended durations.

Repair the stage where each problem occurs

Keep appropriate selectionComplete frame + SRS

The described enrollment frame and random selection are suitable starting points.

Improve participationFollow up with selected students

Use suitable times and response options, especially for the originally selected students who did not answer.

Improve measurementAsk for Tuesday minutes neutrally

Define leaving home, arriving at school and the actual date. Avoid suggesting a preferred duration.

5. Improvements: use follow-up to address the nonresponse concern, and replace the leading threshold item with a neutral numeric travel-time question tied to the actual Tuesday. A brief pilot can check understanding. These changes reduce identified risks; they do not guarantee that every answer will be accurate.

Coverage check: undercoverage is not indicated by the stated frame, because the current enrollment list is complete. Do not add that diagnosis merely because only $48$ students respond. Their missing classmates were selected and then did not answer.

Ready to move on? You should be able to locate a problem in the selection or collection process, explain why it matters for the variable, and distinguish a supported direction from an unsupported guess.

12 · Keep building

Continue learning

Up next: Topic 1.13

Topic 1.13: Experimental Design →

Learn how comparison, random assignment, replication and control help build an experiment that can answer a treatment-effect question.

Previous: Topic 1.11 — Random Sampling · Review the sources of bias · Review contextual explanations · Back to the lesson overview

0000003998 : Abdad Alam Shamim Alam

This is the bestest institution I have ever come to and I love it very much.

Aliki S

Ali

Sly is him

Jitendra Kumar Kumawat

Raahil Hasan

Tina Mudarres

The classes are amazing and my child learnt so much

alisha gadoya

I have experienced a lot of good things, it has taught me so many things and from B/Cs I have gone to an A! This is wonderful and Mavish’s class is awesome.

smasher 123

It’s sensational my kid went. First he only would get C now it’s all A’s really good and recommend

Mosa Al- Samaraie

AMAZING PRICES GREAT TEACHERS STRAIGHT FORWARD LEARNING STEADY PACE IN TUTORING

Cael Dagnelie

I had a great experience learning math with Ms. Mavish. She explains complex topics in a very clear and simple way, which made it easier for me to understand and enjoy the subject. Her patience and dedication really stood out, and she always made sure that everyone in the class was keeping up. I especially appreciated how approachable she was — I never felt afraid to ask questions, and she was always willing to help. Thanks to her teaching, my confidence in math has grown a lot. I’m really thankful for the effort she puts into every lesson!