C1 Activity Models - Longitudinal Models of the MIECHV Effectiveness and Statewide Trends
OUHSC MIECHV External Evaluation Team
- 1 Modeling and Graphical Analysis
- 2 Milestones
- 3 Modeling and Graphical Analysis
- 4 Referrals
- 4.1 Two-rate coding –MIECHV Contracted
- 4.2 Two-rate coding –MIECHV Staffed
- 4.3 Two-rate coding –Has Community Connector
- 4.4 Event: Ad Waves
- 4.5 Event: Ad Waves Lagged
- 4.6 Event: Ad Wave 1
- 4.7 Event: Ad Wave 1 Lagged
- 4.8 Event: Ad Wave 2
- 4.9 Event: Ad Wave 2 Lagged
- 4.10 Event: Referral Restrictions Lifted
- 4.11 Event: ETO Transition
- 4.12 Event: parentPRO Website
- 4.13 Event: Ad Wave 3
- 4.14 Event: Ad Wave 3 Lagged
- 5 Enrollments
- 5.1 Two-rate coding –MIECHV Contracted
- 5.2 Two-rate coding –MIECHV Staffed
- 5.3 Two-rate coding –Has Community Connector
- 5.4 Event: Ad Waves
- 5.5 Event: Ad Waves Lagged
- 5.6 Event: Ad Wave 1
- 5.7 Event: Ad Wave 1 Lagged
- 5.8 Event: Ad Wave 2
- 5.9 Event: Ad Wave 2 Lagged
- 5.10 Event: Referral Restrictions Lifted
- 5.11 Event: ETO Transition
- 5.12 Event: parentPRO Website
- 5.13 Event: Ad Wave 3
- 5.14 Event: Ad Wave 3 Lagged
- 6 Completed Visits Per Need
- 6.1 Two-rate coding –MIECHV Contracted
- 6.2 Two-rate coding –MIECHV Staffed
- 6.3 Two-rate coding –Has Community Connector
- 6.4 Event: Ad Waves
- 6.5 Event: Ad Waves Lagged
- 6.6 Event: Ad Wave 1
- 6.7 Event: Ad Wave 1 Lagged
- 6.8 Event: Ad Wave 2
- 6.9 Event: Ad Wave 2 Lagged
- 6.10 Event: Referral Restrictions Lifted
- 6.11 Event: ETO Transition
- 6.12 Event: parentPRO Website
- 6.13 Event: Ad Wave 3
- 6.14 Event: Ad Wave 3 Lagged
- 7 A tibble: 6 × 3
- 8 Completed Visits Per Nurse Day
- 9 Time & Effort and Infant Need
- 10 Region Tables
- 11 Session Information
1 Modeling and Graphical Analysis
Suitable Referrals coefficient letter beta change p
1 MiechvEver a -0.565 0.57 .001
2 Cohort1 b -0.086 0.92 .292
3 Cohort2 c -0.341 0.71 .007
4 Cohort1:MiechvContracted d 0.022 1.02 .837
5 Cohort2:MiechvContracted e 0.150 1.16 .370Completed Enrollments coefficient letter beta change p
1 MiechvEver a -0.543 0.58 .001
2 Cohort1 b -0.088 0.92 .283
3 Cohort2 c -0.629 0.53 <.001
4 Cohort1:MiechvContracted d -0.007 0.99 .949
5 Cohort2:MiechvContracted e 0.233 1.26 .262Completed Visits coefficient letter beta change p
1 MiechvEver a -0.545 0.58 .001
2 Cohort1 b -0.031 0.97 .314
3 Cohort2 c -0.081 0.92 .096
4 Cohort1:MiechvContracted d 0.045 1.05 .158
5 Cohort2:MiechvContracted e 0.090 1.09 .152
Generalized linear mixed model fit by maximum likelihood (Adaptive Gauss-Hermite Quadrature, nAGQ
= 0) [glmerMod]
Family: poisson ( log )
Formula: EnrollCompletedCount ~ 1 + MiechvEver + Cohort1 + Cohort2 + MiechvContracted:Cohort1 +
MiechvContracted:Cohort2 + (1 + Cohort1 + Cohort2 | RegionID) +
(1 | QuarterF) + offset(log(ReferralCompleteCountMean3Months))
Data: dsRegionMonth2
AIC BIC logLik deviance df.resid
6934.4 7004.6 -3454.2 6908.4 1627
Scaled residuals:
Min 1Q Median 3Q Max
-2.6193 -0.7466 -0.0242 0.5915 5.4015
Random effects:
Groups Name Variance Std.Dev. Corr
QuarterF (Intercept) 2.499e-03 0.049986
RegionID (Intercept) 6.133e-06 0.002477
Cohort1 1.733e-05 0.004163 -1.00
Cohort2 1.617e-02 0.127166 1.00 -1.00
Number of obs: 1640, groups: QuarterF, 28; RegionID, 20
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.010762 0.022027 -0.489 0.6251
MiechvEver -0.001708 0.026335 -0.065 0.9483
Cohort1 -0.020403 0.034877 -0.585 0.5586
Cohort2 -0.396190 0.070551 -5.616 1.96e-08 ***
Cohort1:MiechvContracted 0.001931 0.041758 0.046 0.9631
Cohort2:MiechvContracted 0.201355 0.102142 1.971 0.0487 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) MchvEv Cohrt1 Cohrt2 Ch1:MC
MiechvEver -0.523
Cohort1 -0.632 0.331
Cohort2 -0.296 0.150 0.180
Chrt1:MchvC 0.330 -0.632 -0.537 -0.089
Chrt2:MchvC 0.124 -0.221 -0.073 -0.579 0.123
This report covers the longitudinal models of the MIECHV effectiveness and of statewide trends. It is relevant to Aim 1C (“Evaluate the frequency of referrals coming into EBHV agencies”), Aim 3A (“Continually evaluate the flow of clients served per county”), Aim 3D (“Evaluate effectiveness of new engagement enhancement strategies”), and Aim 5C (“Implement and evaluate system of quality improvements and controls”) of HRSA/ACF D89MC23154 Investigation.
The underlying dataset contains information from 20 lead nurse regions, 11,373 completed referrals, 11,084 completed enrollments, and 241,430 completed visits. It summarizes C1 activity between 2009-01-15 and 2015-12-15. Time and effort data is available from 2012-06-15 to 2015-08-14. The date used for the introduction of MIECHV funds is 2012-06-15.
The processes for gathering and reshaping the C1 data was described in C1 Activity Methods. Information about county and region demographics is available in OSDH Maps.
2 Milestones
One of this investigation’s goals is to assess if C1 performance changes following significant events in Oklahoma’s home visiting programs. The following dates are of particular interest. A more detailed is found on our project’s interactive timeline.
| Start Date | Stop Date | Description | |
|---|---|---|---|
| Cohort 0 | Jan 01, 2009 | Jun 14, 2012 | Comparison period before any MIECHV activities. |
| Community Connector | May 01, 2012 | Present | Community Connector,Community Connector contracts start. (Six counties in Summer & Fall 2012. Comanche stops July 2013. Garfield stops Sept 2014. Kay & Muskogee stops Sept 2015.) |
| Cohort 1 or 2 | Jun 15, 2012 | Present | Program funds are spent in the counties. |
| Cohort 1 | Jun 15, 2012 | Dec 31, 2014 | Program funds start being spent in the counties. |
| Cohort 2 | Jan 01, 2015 | Present | Program funds continue being spent in the counties. |
| Ad Waves | Apr 01, 2014 | Jun 30, 2015 | parentPRO ad campaigns in 2014 and 2015 |
| Ad Waves Lagged | Jun 01, 2014 | Jun 30, 2015 | parentPRO ad campaigns in 2014 and 2015 |
| Ad Wave 1 | Apr 01, 2014 | Jun 30, 2014 | parentPRO ad campaign concentrated in May. |
| Ad Wave 1 Lagged | Jun 01, 2014 | Jun 30, 2014 | parentPRO ad campaign concentrated in May. |
| Ad Wave 2 | Sep 01, 2014 | Oct 30, 2014 | parentPRO ad campaign concentrated in late Sept. |
| Ad Wave 2 Lagged | Oct 01, 2014 | Oct 30, 2014 | parentPRO ad campaign concentrated in late Sept. |
| Referral Restrictions Lifted | May 23, 2014 | Present | Substantial restrictions lifted for inter-agency collaboration. |
| ETO Transition | Jan 01, 2015 | Apr 30, 2015 | Staff transitions from OCAPPA data system to ETO (deployed Jan 1). |
| parentPRO Website | Apr 02, 2015 | Present | The state’s umbrella website is deployed. |
| Ad Wave 3 | May 01, 2015 | Jun 30, 2015 | parentPRO ad campaign concentrated in late May. |
| Ad Wave 3 Lagged | Jun 01, 2015 | Jun 30, 2015 | parentPRO ad campaign concentrated in late May. |
EraWebsite
EraLifted FALSE TRUE
FALSE 1440 126
TRUE 60 54 HasConnector
EraLifted FALSE TRUE
FALSE 1438 128
TRUE 28 86 HasConnector
EraWebsite FALSE TRUE
FALSE 1322 178
TRUE 144 363 Modeling and Graphical Analysis
The outcomes of interest involve the referrals, enrollments, and visits of a C1 region. The MIECHV treatment is operationalized in three ways. In Model 1, a region’s treatment effect reflects the time period when at least one county was under contract with OSDH; the corresponding coefficients are 1a-1g. In Model 2, the treatment effect reflects when a region had at least one MIECHV-funded nurse on staff; the corresponding coefficients are 2a-2g. In Model 3, the treatment effect reflects when a region had at least one ParentPRO community connector.
Models 1, 2, and 3 examine change over time, where clients receiving services are divided into three cohorts. “Cohort 0” was collected from Jan 1 2009 until MIECHV services started around June 2012. “Cohort 1” describes the clients receiving services during the first 19 months of MIECHV (June 1, 2012 until Dec 31, 2014). “Cohort 2” describes the clients receiving services in 2015. Note that many cohort 1 and 2 clients reside in non-MIECHV counties. In that sense, “cohort” can also be conceptualized as a phase or era.
Models 4 through 7 examine specific events related to the service, and combine cohorts 1 and 2 into a single uniform period. Whereas the first three models define MIECHV differently from each other, these last four models evalute the event’s explanatory power after MIECHV has been included in a model. In a sense, these models estimate how much their event explains beyond what the MIECHV variable is already explaining.
A two-level Poisson family is used in each model; the outcome at each month is nested within region. The Poisson offset is the region’s number of infants in need, which help make the regions’ estimates more comparable. Level 1 estimates an intercept and a lag-3 autocorrelation for each region. Level 2 contains two classes of variables: the first attempts to partial out variability unrelated to MIECHV events, while the second addresses the investigation’s hypotheses. The first class estimates the overall level for each quarter year (to remove seasonal and secular trends) and a lag-3 autocorrelation (to control for longitudinal dependencies). The second class estimates a coefficient for the following seven comparisons.
| Coefficient | Label | Description |
|---|---|---|
| a | 0 - NvM | non-MIECHV vs MIECHV at Cohort 0 |
| b | 0v1 - N | Cohort 0vs1 among NonMIECHV |
| c | 0v2 - N | Cohort 0vs2 among NonMIECHV |
| d | 0v1 - NvM | Cohort 0vs1 - non-MIECHV vs MIECHV |
| e | 0v2 - NvM | Cohort 0vs2 - non-MIECHV vs MIECHV |
| f | 1v2 - N | Cohort 1vs2 among NonMIECHV |
| g | 1v2 - NvM | Cohort 1vs2 - non-MIECHV vs MIECHV |
The change represents the proportion difference attributed to each effect, beyond the influence of the model’s other terms. The change value is calculated by \(change = e^{est}\). For example, a value of 1.26 for d indicates that the effect contributed to a 26% increase in the outcome. The main text contains only the salient coefficients listed above; comprehensive model output is available in the appendix and online.
Coefficient a estimates the initial difference between the MIECHV and non-MIECHV regions, and services to control for existing differences unrelated to the MIECHV intervention; a value under 1.00 indicates the MIECHV regions had a lower initial value. Coefficient b estimates how the non-MIECHV regions changed between cohorts 0 and 1; a value under 1.00 indicates the cohort 1 had a lower value than cohort 0. Coefficient c estimates how the non-MIECHV regions changed between cohorts 1 and 2; a value above 1.00 indicates the cohort 2 had a higher value than cohort 1.
Coefficients for d and e are the most relevant to the investigation –they compare how MIECHV changed versus how non-MIECHV changed. A value above 1.00 indicates that the 6 MIECHV regions are increasing more than the 14 comparison regions are increasing (or alternatively, that they are decreasing less). Coefficient d compares this ‘difference of change’ between cohorts 0 and 1, while e compares between 0 and 2. Arguably this is more important than comparing non-MIECHV against MIECHV for a given cohort, but because this considers where they two groups started from (and therefore, if the MIECHV intervention altered the natural trajectory, represented by the comparison regions).
Coefficient f estimates how non-MIECHV regions changed between cohorts 1 and 2, while g estimates how MIECHV regions changed over this same time period, relative to the non-MIECHV regions. A value greater than 1.00 indicates cohort 2 was greater than cohort 1. These point estimates can be calculated from the previous coefficients (f = c/b and g = e/d), but the p-values cannot. Because g is relative, the absolute change for MIECHV regions is found by multiplying f and g. (Note: because the Poisson family uses a loglinear link function, the change coefficients are multiplied and divided, instead of summed and subtracted.)
MIECHV Contracted
- Model 1: MIECHV Contracted: This indicator variable reflects the months each region had at least one MIECHV county under contract. The two formula grant counties (ie, Kay and Garfield) started in May 2011, while the four competition grant counties (ie, Comanche, Muskogee, Oklahoma, and Tulsa) started in October 2011. Coefficients a-g are reported for Model 1.
MIECHV Staffed
- Model 2: MIECHV Nurses Staffed: Some counties hired MIECHV-funded nurses in April 2012, while some could not fill it reliably until the summer of 2014. The model uses the
MiechvStaffedvariable, which is 1 during any month the region has at least one MIECHV-funded nurse, and 0 otherwise. Coefficients a-g are reported for Model 2.
Community Connector
- Model 3: ParentPRO Community Connector in a county. A third way to operationalize MIECHV is the time and location that employeed a ParentPRO Community Connector. The two urban regions started in May 2012, while the remaining four regions started 2 years later in the fall of 2014. The model accommodates the start date of each region.
Isolated Events
Additionally, a model was run for each of the four events to estimate its correspondence with each outcome. The change associated with the event is the only coefficient listed in the main text. Although the other coefficients in the event models differ from each other and from the three-cohort model, they are similar enough to exclude in the main text. Comprehensive model output is available in the appendix and online. A model was run for each of the following events:
- Model 4: First marketing wave. The model accounts for the seven months of ParentPRO advertisements in 2014 across the state. Although residents in MIECHV areas were the focus of this campaign, practically the whole state was exposed (particularly through the day-time television commercials brought in the state’s substantial populations centers). The campaign included internet, television, radio, and transit advertisements.
- Model 5: Referral Restrictions Lifted. Substantial interagency obstacles were removed in May 2014 for the six MIECHV regions in order to make referrals more time efficient.
- Model 6: ParentPRO website. A client-focused website was deployed in April 2015. Like the two marketing waves, this was deployed state-wide.
- Model 7: Second marketing wave. This period during 2015 is handled similarly to the first marketing wave.
A Poisson family was chosen because it typically models counts well, such as the count of referrals in a month. For additional assurance, a conventional model with a Gaussian family was run on an equivalent outcome. For instance, when the Poisson outcome and offset were ‘Referral Count’ and ‘Infant Need Count’ the accompanying Gaussian outcome was simply ‘Referral Count per Infants in Need’. The two approaches lead to comparable conclusions, and only the Poisson is reported.
A longitudinal graph accompanies each outcome. A colored line follows the monthly trajectory of a single region. The six MIECHV regions are a thick colored line, while the fourteen comparison regions are thin. A region’s ID marks its line. The two groups of regions are each summarized by a black loess curve; the MIECHV group is thick, while the comparison group is thin. A faint gray band marks the standard error of each loess curve. Notable dates in MIECHV’s development are marked by faint vertical lines; the start of the three cohorts are coral (pinkish-orange), while the six events are tan.
4 Referrals
We begin by assessing how the MIECHV program affects the number of suitable referrals. The first section addresses the three-cohort model; the three years before MIECHV is compared to cohort 1 (ie, the first 19 months of MIECHV in the state) and to cohort 2 (ie, the next 24 months) of the service. The second section examines the correspondence between an event and referral counts; these four events are analyzed in separate models.
Three cohorts
As seen in 1a, the MIECHV regions started with fewer referrals (per infants in need); in Figure zzz, notice how the thick black summary line is beneath the thin line. This deficit doesn’t necessary indicate that these regions’ personnel are underperforming. An equally likely explanation is that the regions’ environments had client populations more difficult to reach; recall the MIECHV regions are partially composed of the state’s two large urban areas (encompassing Tulsa and Oklahoma City).
Coefficients 1b and 1c are significantly under 1.00, indicating a comparison regions’ trend of fewer and fewer suitable referrals occurring. However coefficients 1d and 1e are significantly and progressively over 1.00, indicating that the six MIECHV regions are outperforming the fourteen comparison regions. Furthermore, 1c \(\times\) 1e produces a number over 1.00, indicating MIECHV regions observed a small improvement in overall referrals between cohorts 0 and 2. This pattern is evident in Figure zzz; notice how the thin black line declines during the seven year period, while the thick line slightly rises.
Comparing cohorts 1 and 2, the 2% decrease for non-MIECHV regions is not significant, while the relative 10% increase for MIECHV regions is significant (ie, 1f=0.98 and 1g=1.10). The absolute increase for MIECHV regions was 8% (ie, 1f \(\times\) 1g = 1.08).
Model 2’s estimates corresponded very closely to Model 1, suggesting that the two operationalizations of the MIECHV treatment are comparable for suitable referrals. Model 3’s estimates also closely correspond to Model 1; the one visible difference is that cohort 1’s level was smaller, which weakened 3d (which estimates MIECHV’s differential gain over non-MIECHV regions between cohorts 0 and 1) and strengthened 3g (the differential gain between cohorts 1 and 2).
MIECHV Contracted MIECHV Staffed Has Connector
Change p Change p Change p
0 - NvM : 1a 0.74 <.001 2a 0.78 .001 3a 0.77 .002
0v1 - N : 1b 0.99 .847 2b 1.01 .786 3b 1.03 .534
0v2 - N : 1c 0.90 .012 2c 0.93 .103 3c 0.92 .055
0v1 - NvM : 1d 1.14 <.001 2d 1.12 <.001 3d 1.04 .228
0v2 - NvM : 1e 1.26 <.001 2e 1.18 <.001 3e 1.22 <.001
1v2 - N : 1f 0.91 .055 2f 0.92 .106 3f 0.90 .031
1v2 - NvM : 1g 1.11 .003 2g 1.06 .128 3g 1.17 <.001Individual Events
Similarly, Model 4 associated the first advertising wave (beginning in April 2014) with a 14% increase state-wide. Model 5 indicates that streamlining the referral process in MIECHV counties (which started in May 2014, just one month after the first ads appeared) corresponded with a 6% increase. Collinearity is not a concern because these predictors exist in separate models. Models 6 has a large p-value, showing no evidence of an effect for state website (April 2015). However the second advertising wave (Summer 2015) corresponded with a 15% decrease state-wide.
Event Change p
4 AdWaveAny 0.96 .319
5 AdWave1 1.03 .607
6 AdWave2 1.03 .572
7 EraLifted 1.08 .030
8 EraWebsite 0.94 .210
9 AdWave3 0.86 .014
Response variable: ReferralCompleteCount; Offset variable: InfantNeedCountForTime
4.1 Two-rate coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.565 0.57 .001
2 Cohort1 b -0.086 0.92 .292
3 Cohort2 c -0.341 0.71 .007
4 Cohort1:MiechvContracted d 0.022 1.02 .837
5 Cohort2:MiechvContracted e 0.150 1.16 .370
Generalized linear mixed model fit by maximum likelihood (Adaptive Gauss-Hermite Quadrature, nAGQ
= 0) [glmerMod]
Family: poisson ( log )
Formula: ReferralCompleteCount ~ 1 + MiechvEver + Cohort1 + Cohort2 +
MiechvContracted:Cohort1 + MiechvContracted:Cohort2 + Lag1 +
Lag2 + Lag3 + (1 + Cohort1 + Cohort2 + Lag1 + Lag2 + Lag3 |
RegionID) + (1 | QuarterF) + offset(log(InfantNeedCountForTime))
Data: d
AIC BIC logLik deviance df.resid
7975.8 8142.9 -3956.9 7913.8 1589
Random effects:
Groups Name Variance Std.Dev. Corr
QuarterF (Intercept) 0.0175915 0.13263
RegionID (Intercept) 0.1601920 0.40024
Cohort1 0.0364567 0.19094 -0.68
Cohort2 0.1037230 0.32206 -0.51 0.26
Lag1 0.0011825 0.03439 -0.60 0.18 -0.24
Lag2 0.0024474 0.04947 -0.43 0.11 -0.22 0.89
Lag3 0.0002638 0.01624 -0.45 -0.26 0.70 0.23 0.21
Number of obs: 1620, groups: QuarterF, 27; RegionID, 20
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.68099 0.11235 -23.862 < 2e-16 ***
MiechvEver -0.56500 0.16329 -3.460 0.000540 ***
Cohort1 -0.08571 0.08139 -1.053 0.292324
Cohort2 -0.34102 0.12569 -2.713 0.006665 **
Lag1 0.03210 0.01122 2.862 0.004215 **
Lag2 0.05088 0.01416 3.594 0.000326 ***
Lag3 0.03511 0.00880 3.990 6.61e-05 ***
Cohort1:MiechvContracted 0.02151 0.10453 0.206 0.836976
Cohort2:MiechvContracted 0.14981 0.16713 0.896 0.370073
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
4.1.1 Increment coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.627 0.53 <.001
2 Cohort1Or2 b -0.084 0.92 .157
3 Cohort2 c -0.195 0.82 .028
4 Cohort1Or2:MiechvContracted d 0.048 1.05 .258
5 Cohort2:MiechvContracted e 0.209 1.23 .002
AIC BIC logLik deviance df.resid
8049.980 8157.783 -4004.990 8009.980 1600.000
4.2 Two-rate coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.573 0.56 <.001
2 Cohort1 b -0.088 0.92 .244
3 Cohort2 c -0.348 0.71 .003
4 Cohort1:MiechvStaffed d 0.045 1.05 .430
5 Cohort2:MiechvStaffed e 0.195 1.22 .070
AIC BIC logLik deviance df.resid
7973.015 8140.110 -3955.507 7911.015 1589.000
4.2.1 Increment coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.607 0.54 <.001
2 Cohort1Or2 b -0.080 0.92 .174
3 Cohort2 c -0.199 0.82 .022
4 Cohort1Or2:MiechvStaffed d 0.048 1.05 .242
5 Cohort2:MiechvStaffed e 0.279 1.32 <.001
AIC BIC logLik deviance df.resid
8039.298 8147.102 -3999.649 7999.298 1600.000
4.3 Two-rate coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.573 0.56 <.001
2 Cohort1 b -0.067 0.94 .383
3 Cohort2 c -0.389 0.68 .001
4 Cohort1:HasConnector d -0.051 0.95 .497
5 Cohort2:HasConnector e 0.384 1.47 .006
AIC BIC logLik deviance df.resid
7969.709 8136.804 -3953.854 7907.709 1589.000
4.3.1 Increment coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.620 0.54 <.001
2 Cohort1Or2 b -0.061 0.94 .296
3 Cohort2 c -0.234 0.79 .008
4 Cohort1Or2:HasConnector d -0.002 1.00 .967
5 Cohort2:HasConnector e 0.315 1.37 <.001
AIC BIC logLik deviance df.resid
8040.757 8148.561 -4000.379 8000.757 1600.000
4.4 Event: Ad Waves
Event Description: parentPRO ad campaigns in 2014 and 2015 - Apr 01, 2014
coefficient letter beta change p
1 AdWaveAny h -0.124 0.88 .002
2 Cohort1Or2 i -0.050 0.95 .016
AIC BIC logLik deviance df.resid
8165.279 8251.522 -4066.640 8133.279 1604.000
4.5 Event: Ad Waves Lagged
Event Description: parentPRO ad campaigns in 2014 and 2015 - Jun 01, 2014
coefficient letter beta change p
1 AdWaveAnyLagged h -0.190 0.83 .001
2 Cohort1Or2 i -0.056 0.95 .005
AIC BIC logLik deviance df.resid
8164.546 8250.788 -4066.273 8132.546 1604.000
4.6 Event: Ad Wave 1
Event Description: parentPRO ad campaign concentrated in May. - Apr 01, 2014
coefficient letter beta change p
1 AdWave1 h -0.053 0.95 .348
2 Cohort1Or2 i -0.064 0.94 .001
AIC BIC logLik deviance df.resid
8174.487 8260.730 -4071.244 8142.487 1604.000
4.7 Event: Ad Wave 1 Lagged
Event Description: parentPRO ad campaign concentrated in May. - Jun 01, 2014
coefficient letter beta change p
1 AdWave1Lagged h -0.112 0.89 .254
2 Cohort1Or2 i -0.067 0.94 .001
AIC BIC logLik deviance df.resid
8194.899 8281.142 -4081.449 8162.899 1604.000
4.8 Event: Ad Wave 2
Event Description: parentPRO ad campaign concentrated in late Sept. - Sep 01, 2014
coefficient letter beta change p
1 AdWave2 h -0.185 0.83 .008
2 Cohort1Or2 i -0.059 0.94 .003
AIC BIC logLik deviance df.resid
8167.821 8254.064 -4067.910 8135.821 1604.000
4.9 Event: Ad Wave 2 Lagged
Event Description: parentPRO ad campaign concentrated in late Sept. - Oct 01, 2014
coefficient letter beta change p
1 AdWave2Lagged h -0.242 0.79 .017
2 Cohort1Or2 i -0.062 0.94 .002
AIC BIC logLik deviance df.resid
8169.064 8255.307 -4068.532 8137.064 1604.000
4.10 Event: Referral Restrictions Lifted
Event Description: Substantial restrictions lifted for inter-agency collaboration. - May 23, 2014
coefficient letter beta change p
1 EraLifted h -0.092 0.91 .019
2 Cohort1Or2 i -0.051 0.95 .016
AIC BIC logLik deviance df.resid
8169.783 8256.025 -4068.891 8137.783 1604.000
4.11 Event: ETO Transition
Event Description: Staff transitions from OCAPPA data system to ETO (deployed Jan 1). - Jan 01, 2015
coefficient letter beta change p
1 EtoTransition h 0.054 1.06 .279
2 Cohort1Or2 i -0.072 0.93 <.001
AIC BIC logLik deviance df.resid
8174.022 8260.265 -4071.011 8142.022 1604.000
4.12 Event: parentPRO Website
Event Description: The state’s umbrella website is deployed. - Apr 02, 2015
coefficient letter beta change p
1 EraWebsite h -0.108 0.90 .004
2 Cohort1Or2 i -0.050 0.95 .015
AIC BIC logLik deviance df.resid
8166.942 8253.185 -4067.471 8134.942 1604.000
4.13 Event: Ad Wave 3
Event Description: parentPRO ad campaign concentrated in late May. - May 01, 2015
coefficient letter beta change p
1 AdWave3 h -0.120 0.89 .102
2 Cohort1Or2 i -0.063 0.94 .002
AIC BIC logLik deviance df.resid
8172.417 8258.660 -4070.208 8140.417 1604.000
4.14 Event: Ad Wave 3 Lagged
Event Description: parentPRO ad campaign concentrated in late May. - Jun 01, 2015
coefficient letter beta change p
1 AdWave3Lagged h -0.184 0.83 .080
2 Cohort1Or2 i -0.064 0.94 .001
AIC BIC logLik deviance df.resid
8171.929 8258.172 -4069.965 8139.929 1604.000
5 Enrollments
The next batch of models assessed MIECHV’s effect on completed enrollments, which occurs after the client’s first completed visit.
Three cohorts
As seen in 1a, the MIECHV regions started with 36% fewer enrollments (per infants in need); in Figure zzz, notice how the thick black summary line is beneath the thin line. Coefficients 1b and 1c are significantly less than 1.00, indicating the comparison regions’ trend of fewer completed enrollments. Coefficients 1d and 1e are greater than 1.00, but only 1e is significant; this indicates that the six MIECHV regions are comparatively outgaining the fourteen comparison regions between cohorts 0 and 2. However, 1b \(\times\) 1d produces a value under 1.00 (and 1c \(\times\) 1e do too), indicating MIECHV regions observed a decline in overall enrollments (but a much less steep decline than comparison regions). This pattern is evident in Figure zzz; notice how the thin black line descends sharply during the seven year period, while the thick line declines more gradually. The groups’ enrollment rates are almost equal by December 2015.
Comparing cohorts 1 and 2, the 11% decrease for non-MIECHV regions is not significant, and neither is the relative 9% increase for MIECHV regions (ie, 1f=1.03 and 1g=1.09).
The esimates from Models 2 and 3 corresponded closely to Model 1, suggesting that the three operationalizations of the MIECHV treatment are comparable for completed enrollments. The only notable difference involves coefficient g. The Model 2 and 3 estimates are about 5%, at which point coefficients become significant; this suggests MIECHV regions did relatively better than non-MIECHV regions between cohorts 1 and 2.
MIECHV Contracted MIECHV Staffed Has Connector
Change p Change p Change p
0 - NvM : 1a 0.65 <.001 2a 0.65 <.001 3a 0.67 <.001
0v1 - N : 1b 0.98 .776 2b 0.99 .910 3b 1.01 .880
0v2 - N : 1c 0.80 <.001 2c 0.80 <.001 3c 0.82 .001
0v1 - NvM : 1d 1.06 .216 2d 1.05 .303 3d 1.00 .936
0v2 - NvM : 1e 1.15 .004 2e 1.19 <.001 3e 1.11 .028
1v2 - N : 1f 0.81 .005 2f 0.80 .002 3f 0.81 .004
1v2 - NvM : 1g 1.09 .144 2g 1.13 .040 3g 1.12 .057Individual Events
Model 4 associated the first advertising wave (beginning in April 2014) with a 20% increase; although this is similar to how referral increased during the first ads, the enrollment trajectories are so variable that it was still not significant. Models 5 and 6 have large p-values, showing no evidence of an effect for the streamlining (which started in May 2014) or the deployment of the state website (April 2015). Model 7 indicates the second advertising wave (Summer 2015) corresponded with a 22% decline.
Event Change p
4 AdWaveAny 0.98 .797
5 AdWave1 0.89 .307
6 AdWave2 1.20 .151
7 EraLifted 1.09 .145
8 EraWebsite 0.72 <.001
9 AdWave4 0.91 .419
Response variable: EnrollCompletedCount; Offset variable: InfantNeedCountForTime
5.1 Two-rate coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.543 0.58 .001
2 Cohort1 b -0.088 0.92 .283
3 Cohort2 c -0.629 0.53 <.001
4 Cohort1:MiechvContracted d -0.007 0.99 .949
5 Cohort2:MiechvContracted e 0.233 1.26 .262
Generalized linear mixed model fit by maximum likelihood (Adaptive Gauss-Hermite Quadrature, nAGQ
= 0) [glmerMod]
Family: poisson ( log )
Formula: EnrollCompletedCount ~ 1 + MiechvEver + Cohort1 + Cohort2 + MiechvContracted:Cohort1 +
MiechvContracted:Cohort2 + Lag1 + Lag2 + Lag3 + (1 + Cohort1 +
Cohort2 + Lag1 + Lag2 + Lag3 | RegionID) + (1 | QuarterF) +
offset(log(InfantNeedCountForTime))
Data: d
AIC BIC logLik deviance df.resid
7844.7 8011.8 -3891.4 7782.7 1589
Random effects:
Groups Name Variance Std.Dev. Corr
QuarterF (Intercept) 0.0183329 0.13540
RegionID (Intercept) 0.1545976 0.39319
Cohort1 0.0357394 0.18905 -0.68
Cohort2 0.1566972 0.39585 -0.66 0.40
Lag1 0.0012328 0.03511 -0.55 0.15 0.18
Lag2 0.0025087 0.05009 -0.42 0.27 0.23 0.91
Lag3 0.0002687 0.01639 -0.33 -0.44 0.40 0.26 -0.05
Number of obs: 1620, groups: QuarterF, 27; RegionID, 20
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.702975 0.111680 -24.203 < 2e-16 ***
MiechvEver -0.542869 0.166546 -3.260 0.001116 **
Cohort1 -0.087938 0.081835 -1.075 0.282564
Cohort2 -0.628509 0.145089 -4.332 1.48e-05 ***
Lag1 0.031831 0.011379 2.797 0.005151 **
Lag2 0.054834 0.014280 3.840 0.000123 ***
Lag3 0.036316 0.009244 3.929 8.54e-05 ***
Cohort1:MiechvContracted -0.006558 0.103070 -0.064 0.949271
Cohort2:MiechvContracted 0.232520 0.207472 1.121 0.262402
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
5.1.1 Increment coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.559 0.57 <.001
2 Cohort1Or2 b -0.092 0.91 .126
3 Cohort2 c -0.428 0.65 <.001
4 Cohort1Or2:MiechvContracted d 0.055 1.06 .193
5 Cohort2:MiechvContracted e 0.302 1.35 <.001
AIC BIC logLik deviance df.resid
7914.816 8022.620 -3937.408 7874.816 1600.000
5.2 Two-rate coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.545 0.58 <.001
2 Cohort1 b -0.096 0.91 .210
3 Cohort2 c -0.611 0.54 <.001
4 Cohort1:MiechvStaffed d 0.028 1.03 .619
5 Cohort2:MiechvStaffed e 0.200 1.22 .100
AIC BIC logLik deviance df.resid
7843.376 8010.472 -3890.688 7781.376 1589.000
5.2.1 Increment coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.523 0.59 <.001
2 Cohort1Or2 b -0.083 0.92 .165
3 Cohort2 c -0.393 0.67 <.001
4 Cohort1Or2:MiechvStaffed d 0.041 1.04 .311
5 Cohort2:MiechvStaffed e 0.320 1.38 <.001
AIC BIC logLik deviance df.resid
7912.869 8020.673 -3936.434 7872.869 1600.000
5.3 Two-rate coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.508 0.60 <.001
2 Cohort1 b -0.071 0.93 .355
3 Cohort2 c -0.631 0.53 <.001
4 Cohort1:HasConnector d -0.066 0.94 .376
5 Cohort2:HasConnector e 0.339 1.40 .047
AIC BIC logLik deviance df.resid
7841.016 8008.112 -3889.508 7779.016 1589.000
5.3.1 Increment coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.543 0.58 <.001
2 Cohort1Or2 b -0.069 0.93 .249
3 Cohort2 c -0.460 0.63 <.001
4 Cohort1Or2:HasConnector d 0.003 1.00 .941
5 Cohort2:HasConnector e 0.399 1.49 <.001
AIC BIC logLik deviance df.resid
7906.333 8014.137 -3933.167 7866.333 1600.000
5.4 Event: Ad Waves
Event Description: parentPRO ad campaigns in 2014 and 2015 - Apr 01, 2014
coefficient letter beta change p
1 AdWaveAny h -0.138 0.87 .001
2 Cohort1Or2 i -0.079 0.92 <.001
AIC BIC logLik deviance df.resid
8073.262 8159.505 -4020.631 8041.262 1604.000
5.5 Event: Ad Waves Lagged
Event Description: parentPRO ad campaigns in 2014 and 2015 - Jun 01, 2014
coefficient letter beta change p
1 AdWaveAnyLagged h -0.182 0.83 .003
2 Cohort1Or2 i -0.088 0.92 <.001
AIC BIC logLik deviance df.resid
8075.605 8161.848 -4021.802 8043.605 1604.000
5.6 Event: Ad Wave 1
Event Description: parentPRO ad campaign concentrated in May. - Apr 01, 2014
coefficient letter beta change p
1 AdWave1 h -0.014 0.99 .808
2 Cohort1Or2 i -0.098 0.91 <.001
AIC BIC logLik deviance df.resid
8084.725 8170.968 -4026.362 8052.725 1604.000
5.7 Event: Ad Wave 1 Lagged
Event Description: parentPRO ad campaign concentrated in May. - Jun 01, 2014
coefficient letter beta change p
1 AdWave1Lagged h -0.083 0.92 .400
2 Cohort1Or2 i -0.097 0.91 <.001
AIC BIC logLik deviance df.resid
8084.062 8170.305 -4026.031 8052.062 1604.000
5.8 Event: Ad Wave 2
Event Description: parentPRO ad campaign concentrated in late Sept. - Sep 01, 2014
coefficient letter beta change p
1 AdWave2 h -0.164 0.85 .020
2 Cohort1Or2 i -0.091 0.91 <.001
AIC BIC logLik deviance df.resid
8079.138 8165.381 -4023.569 8047.138 1604.000
5.9 Event: Ad Wave 2 Lagged
Event Description: parentPRO ad campaign concentrated in late Sept. - Oct 01, 2014
coefficient letter beta change p
1 AdWave2Lagged h -0.210 0.81 .039
2 Cohort1Or2 i -0.094 0.91 <.001
AIC BIC logLik deviance df.resid
8080.248 8166.491 -4024.124 8048.248 1604.000
5.10 Event: Referral Restrictions Lifted
Event Description: Substantial restrictions lifted for inter-agency collaboration. - May 23, 2014
coefficient letter beta change p
1 EraLifted h -0.166 0.85 <.001
2 Cohort1Or2 i -0.069 0.93 .001
AIC BIC logLik deviance df.resid
8068.441 8154.683 -4018.220 8036.441 1604.000
5.11 Event: ETO Transition
Event Description: Staff transitions from OCAPPA data system to ETO (deployed Jan 1). - Jan 01, 2015
coefficient letter beta change p
1 EtoTransition h -0.098 0.91 .075
2 Cohort1Or2 i -0.092 0.91 <.001
AIC BIC logLik deviance df.resid
8081.614 8167.857 -4024.807 8049.614 1604.000
5.12 Event: parentPRO Website
Event Description: The state’s umbrella website is deployed. - Apr 02, 2015
coefficient letter beta change p
1 EraWebsite h -0.269 0.76 <.001
2 Cohort1Or2 i -0.064 0.94 .002
AIC BIC logLik deviance df.resid
8042.975 8129.218 -4005.488 8010.975 1604.000
5.13 Event: Ad Wave 3
Event Description: parentPRO ad campaign concentrated in late May. - May 01, 2015
coefficient letter beta change p
1 AdWave3 h -0.281 0.75 .001
2 Cohort1Or2 i -0.091 0.91 <.001
AIC BIC logLik deviance df.resid
8072.509 8158.752 -4020.254 8040.509 1604.000
5.14 Event: Ad Wave 3 Lagged
Event Description: parentPRO ad campaign concentrated in late May. - Jun 01, 2015
coefficient letter beta change p
1 AdWave3Lagged h -0.247 0.78 .033
2 Cohort1Or2 i -0.095 0.91 <.001
AIC BIC logLik deviance df.resid
8079.921 8166.163 -4023.960 8047.921 1604.000
6 Completed Visits Per Need
The first visit outcome is visits per need, which closely resembles the previous model of enrollment per need, as seen in Figure zzz and Table zzz. The MIECHV regions started with lower values, but this gap narrowed as both groups declined during the seven years. The gap closed more quickly in the last 24 months than the first 19. Visit counts appears much more stable than referral or enrollment counts, so the graph’s region trajectories are much calmer. The appendix shows the lag 1 z-score is four times stronger.
Three cohorts
Coefficients 1b and 1c are significantly less than 1.00, indicating the comparison regions’ trend of 6% fewer suitable visits being completed. Coefficients 1d and 1e are significantly greater than 1.00, indicating that the six MIECHV regions are comparatively outperforming the fourteen comparison regions between cohorts 0 and 2. However, 1b \(\times\) 1d produces a value under 1.00 (and 1c \(\times\) 1e do too), indicating MIECHV regions observed a decline in overall enrollments (but a less steep decline than comparison regions).
The esimates from Models 2 and 3 corresponded closely to Model 1, suggesting that the two operationalizations of the MIECHV treatment are comparable for completed visits. The only notable different is Model 2 detects the NvM change later, so that the 2d is not significant, by 2g is.
It is interesting that the model could detect the significance of a 2% difference with this outcome, when larger differences were not detected for some previous outcomes. This is likely to due to the cleaner visits per need signal. When referral or enrollment is the outcome, the trajectories are much noisier, which weakens power. This hints that visit count is a good candidate for future interventions, including the C1 best practice workshops.
MIECHV Contracted MIECHV Staffed Has Connector
Change p Change p Change p
0 - NvM : 1a 0.94 .007 2a 0.94 .009 3a 0.95 .016
0v1 - N : 1b 0.99 .781 2b 1.00 .971 3b 0.99 .856
0v2 - N : 1c 0.96 .178 2c 0.96 .128 3c 0.97 .284
0v1 - NvM : 1d 1.02 .041 2d 1.01 .430 3d 1.02 .121
0v2 - NvM : 1e 1.04 .001 2e 1.05 <.001 3e 1.02 .058
1v2 - N : 1f 0.97 .384 2f 0.96 .217 3f 0.98 .466
1v2 - NvM : 1g 1.01 .295 2g 1.04 .001 3g 1.00 .751Individual Events
Model 4 indicates the first advertising campaign corresponds with a huge 38% increase. Models 6 and 7 suggest 10% and 16% decreases correspond with the website and second advertising campaign since the website was deployed. The low correspondence suggested by some of the models may not be surprising. The efforts associated with these four events were intended to boost the referral and enrollment count; affecting the visits was not their goals. However we do not have a good explanation about the large changes associated with the marketing efforts. Model 5 has a large p-value, suggesting no effect on visits.
Event Change p
4 AdWaveAny 1.08 <.001
5 AdWave1 0.98 .343
6 AdWave2 1.38 <.001
7 EraLifted 1.01 .341
8 EraWebsite 0.90 <.001
9 AdWave3 0.84 <.001
Response variable: VisitCompletedCount; Offset variable: InfantNeedCountForTime
6.1 Two-rate coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.545 0.58 .001
2 Cohort1 b -0.031 0.97 .314
3 Cohort2 c -0.081 0.92 .096
4 Cohort1:MiechvContracted d 0.045 1.05 .158
5 Cohort2:MiechvContracted e 0.090 1.09 .152
Generalized linear mixed model fit by maximum likelihood (Adaptive Gauss-Hermite Quadrature, nAGQ
= 0) [glmerMod]
Family: poisson ( log )
Formula: VisitCompletedCount ~ 1 + MiechvEver + Cohort1 + Cohort2 + MiechvContracted:Cohort1 +
MiechvContracted:Cohort2 + Lag1 + Lag2 + Lag3 + (1 + Cohort1 +
Cohort2 + Lag1 + Lag2 + Lag3 | RegionID) + (1 | QuarterF) +
offset(log(InfantNeedCountForTime))
Data: d
AIC BIC logLik deviance df.resid
15732.9 15900.0 -7835.5 15670.9 1589
Random effects:
Groups Name Variance Std.Dev. Corr
QuarterF (Intercept) 0.003389 0.05822
RegionID (Intercept) 0.365273 0.60438
Cohort1 0.004513 0.06718 -0.05
Cohort2 0.015370 0.12398 -0.25 0.32
Lag1 0.028311 0.16826 -0.28 -0.30 0.21
Lag2 0.002848 0.05337 -0.53 -0.20 -0.20 -0.47
Lag3 0.007945 0.08913 0.04 0.46 -0.25 -0.92 0.58
Number of obs: 1620, groups: QuarterF, 27; RegionID, 20
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.75031 0.16132 -17.049 < 2e-16 ***
MiechvEver -0.54475 0.17011 -3.202 0.00136 **
Cohort1 -0.03130 0.03108 -1.007 0.31399
Cohort2 -0.08096 0.04860 -1.666 0.09574 .
Lag1 0.33050 0.04046 8.169 3.12e-16 ***
Lag2 0.24671 0.01835 13.447 < 2e-16 ***
Lag3 0.10399 0.02366 4.394 1.11e-05 ***
Cohort1:MiechvContracted 0.04485 0.03179 1.411 0.15829
Cohort2:MiechvContracted 0.08968 0.06253 1.434 0.15155
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
6.1.1 Increment coding –MIECHV Contracted
coefficient letter beta change p
1 MiechvEver a -0.642 0.53 .001
2 Cohort1Or2 b -0.002 1.00 .942
3 Cohort2 c -0.003 1.00 .931
4 Cohort1Or2:MiechvContracted d 0.028 1.03 .003
5 Cohort2:MiechvContracted e 0.035 1.04 .014
AIC BIC logLik deviance df.resid
15885.026 15992.830 -7922.513 15845.026 1600.000
6.2 Two-rate coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.522 0.59 <.001
2 Cohort1 b -0.022 0.98 .464
3 Cohort2 c -0.088 0.92 .053
4 Cohort1:MiechvStaffed d 0.019 1.02 .147
5 Cohort2:MiechvStaffed e 0.126 1.13 <.001
AIC BIC logLik deviance df.resid
15709.798 15876.894 -7823.899 15647.798 1589.000
6.2.1 Increment coding –MIECHV Staffed
coefficient letter beta change p
1 MiechvEver a -0.684 0.50 <.001
2 Cohort1Or2 b 0.002 1.00 .927
3 Cohort2 c -0.010 0.99 .790
4 Cohort1Or2:MiechvStaffed d 0.022 1.02 .013
5 Cohort2:MiechvStaffed e 0.063 1.07 <.001
AIC BIC logLik deviance df.resid
15866.847 15974.651 -7913.424 15826.847 1600.000
6.3 Two-rate coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.426 0.65 .003
2 Cohort1 b -0.031 0.97 .315
3 Cohort2 c -0.067 0.93 .146
4 Cohort1:HasConnector d 0.049 1.05 .035
5 Cohort2:HasConnector e 0.052 1.05 .236
AIC BIC logLik deviance df.resid
15731.419 15898.515 -7834.709 15669.419 1589.000
6.3.1 Increment coding –Has Community Connector
coefficient letter beta change p
1 MiechvEver a -0.629 0.53 .001
2 Cohort1Or2 b 0.002 1.00 .952
3 Cohort2 c 0.003 1.00 .929
4 Cohort1Or2:HasConnector d 0.022 1.02 .020
5 Cohort2:HasConnector e 0.025 1.03 .076
AIC BIC logLik deviance df.resid
15893.802 16001.605 -7926.901 15853.802 1600.000
6.4 Event: Ad Waves
Event Description: parentPRO ad campaigns in 2014 and 2015 - Apr 01, 2014
coefficient letter beta change p
1 AdWaveAny h 0.024 1.02 .003
2 Cohort1Or2 i 0.008 1.01 .101
AIC BIC logLik deviance df.resid
16524.990 16611.233 -8246.495 16492.990 1604.000
6.5 Event: Ad Waves Lagged
Event Description: parentPRO ad campaigns in 2014 and 2015 - Jun 01, 2014
coefficient letter beta change p
1 AdWaveAnyLagged h 0.044 1.05 <.001
2 Cohort1Or2 i 0.008 1.01 .075
AIC BIC logLik deviance df.resid
16519.321 16605.564 -8243.661 16487.321 1604.000
6.6 Event: Ad Wave 1
Event Description: parentPRO ad campaign concentrated in May. - Apr 01, 2014
coefficient letter beta change p
1 AdWave1 h 0.009 1.01 .423
2 Cohort1Or2 i 0.011 1.01 .026
AIC BIC logLik deviance df.resid
16532.938 16619.181 -8250.469 16500.938 1604.000
6.7 Event: Ad Wave 1 Lagged
Event Description: parentPRO ad campaign concentrated in May. - Jun 01, 2014
coefficient letter beta change p
1 AdWave1Lagged h 0.000 1.00 .985
2 Cohort1Or2 i 0.011 1.01 .019
AIC BIC logLik deviance df.resid
16533.578 16619.821 -8250.789 16501.578 1604.000
6.8 Event: Ad Wave 2
Event Description: parentPRO ad campaign concentrated in late Sept. - Sep 01, 2014
coefficient letter beta change p
1 AdWave2 h 0.091 1.10 <.001
2 Cohort1Or2 i 0.007 1.01 .138
AIC BIC logLik deviance df.resid
16490.40 16576.64 -8229.20 16458.40 1604.00
6.9 Event: Ad Wave 2 Lagged
Event Description: parentPRO ad campaign concentrated in late Sept. - Oct 01, 2014
coefficient letter beta change p
1 AdWave2Lagged h 0.124 1.13 <.001
2 Cohort1Or2 i 0.008 1.01 .082
AIC BIC logLik deviance df.resid
16491.327 16577.570 -8229.663 16459.327 1604.000
6.10 Event: Referral Restrictions Lifted
Event Description: Substantial restrictions lifted for inter-agency collaboration. - May 23, 2014
coefficient letter beta change p
1 EraLifted h 0.009 1.01 .317
2 Cohort1Or2 i 0.010 1.01 .045
AIC BIC logLik deviance df.resid
16532.587 16618.830 -8250.293 16500.587 1604.000
6.11 Event: ETO Transition
Event Description: Staff transitions from OCAPPA data system to ETO (deployed Jan 1). - Jan 01, 2015
coefficient letter beta change p
1 EtoTransition h 0.081 1.08 <.001
2 Cohort1Or2 i 0.006 1.01 .186
AIC BIC logLik deviance df.resid
16477.78 16564.02 -8222.89 16445.78 1604.00
6.12 Event: parentPRO Website
Event Description: The state’s umbrella website is deployed. - Apr 02, 2015
coefficient letter beta change p
1 EraWebsite h -0.012 0.99 .120
2 Cohort1Or2 i 0.013 1.01 .008
AIC BIC logLik deviance df.resid
16531.186 16617.429 -8249.593 16499.186 1604.000
6.13 Event: Ad Wave 3
Event Description: parentPRO ad campaign concentrated in late May. - May 01, 2015
coefficient letter beta change p
1 AdWave3 h -0.038 0.96 .010
2 Cohort1Or2 i 0.012 1.01 .009
AIC BIC logLik deviance df.resid
16526.98 16613.22 -8247.49 16494.98 1604.00
6.14 Event: Ad Wave 3 Lagged
Event Description: parentPRO ad campaign concentrated in late May. - Jun 01, 2015
coefficient letter beta change p
1 AdWave3Lagged h -0.005 0.99 .793
2 Cohort1Or2 i 0.011 1.01 .018
AIC BIC logLik deviance df.resid
16533.510 16619.752 -8250.755 16501.510 1604.000
7 A tibble: 6 × 3
MiechvEver Cohort visit_mean
8 Completed Visits Per Nurse Day
The second visit outcome is visits per nurse day, which can be interpreted as a type of efficiency: for each day a nurse is working, how many visits are being completed?
This outcome is examined during a shorter timeframe than the previous outcomes. June 2012 is the starting point (instead of January 2009) because OSDH switched to a new “Time & Effort” database system, and the values from before were not directly comparable.
Three cohorts
The switch occurred roughly at the beginning of cohort 1, so the only time comparison is cohort 1 vs cohort 2; therefore coefficients have a slightly different interpretation. Only coefficients 1a, 1c, and 1e are meaningful.
Coefficient 1a describes the initial differences between MIECHV and non-MIECHV regions (however this is applies to cohort 1 differences, instead of cohort 0); the MIECHV regions completed slightly, and nonsignificantly, fewer visits during cohort 1. Coefficient 1c describes the drop between cohorts 1 and 2 for non-MIECHV regions; cohort 2 had nonsignificantly fewer visits. Coefficient 1e describes the increase of MIECHV regions, relative to non-MIECHV regions; they improved
The MIECHV and non-MIECHV regions had comparable levels during cohort 1 (note 1a is close to 1.00 with a nonsignificant p-value). Similarly, 1c indicates the non-MIECHV regions had similar levels during cohort 1 and cohort 2. However the MIECHV regions had a significantly 4% better productivity than the non-MIECHV regions. This is reflected by the gap between the two black loess lines that emerges soon before cohort 2 begins.
Again the esimates from Models 2 and 3 corresponded closely to Model 1, but their coefficients are slightly closer to 1.00. For Model 2, this might be because its “staffing” component is partially absorbed by the outcome’s denominator (ie, count of working days).
MIECHV Contracted MIECHV Staffed MIECHV Staffed
Change p Change p Change p
0 - NvM : 1a 0.97 .153 2a 0.97 .059 3a 0.96 .102
1v2 - N : 1c 0.98 .567 2c 0.99 .789 3c 0.99 .679
1v2 - NvM : 1e 1.04 .005 2e 1.03 .053 3e 1.03 .024Individual Events
Models 4 shows the first marketing campaign corresponds with a large 28% increase in visits, yet again Model 7 shows the second campaign corresponds with a large 15% decrease. Model 5 indicates a significant 4% increase following the streamlined referral process, while Moddel 6d show almost no change corresponds with the website deployment.
Event Change p
4 AdWaveAny 1.11 <.001
5 AdWave1 1.04 .143
6 AdWave2 1.28 <.001
7 EraLifted 1.04 .002
8 EraWebsite 0.99 .777
9 AdWave3 1.03 .218
9 Time & Effort and Infant Need
The staffing and demographic trends of each region are shown below. They are not considered outcomes of the investigation, but provide context for the previous sections.
The “WIC Estimate of Infant Need” graph shows how many infants in a region are in need. Our analyses uses this data source, instead of the US Census, because it better reflects the population served by home visiting programs. The Oklahoma and Tulsa regions (19 and 20) have much more need, despite covering a smaller geographical area. The six MIECHV regions average roughly 2,500 infants per year, while the 14 comparison regions average roughly 1,000. The actual yearly WIC estimates were smoothed so the denominator would be more stable in the current analyses. The OSDH Maps document displays the location of the region and counties, and describes the smoothing process of the need estimates.
The “Paid Nurse Days” graph represents the sum of days worked by nurses. This procedure is described in the “Estimating Nurse Days” section of the C1 Activity Methods document online. The measure begins in 2012, corresponding to OSDH’s transition to their new “Time and Effort” data system.
The “Paid Nurse Days per Need” graph divides the values of the second graph by the first. There is large variability between regions’ staffing levels, relative to their population in need. It is common to see a factor of 2x or 3x difference between regions. Relevant to the previous outcomes, the thin loess line is between 30% and 50% higher than the thick line, indicating non-MIECHV regions tend to have more staff per need, compared to MIECHV regions. The Poisson models accommodate the initial differences with coefficient a.
All online documents can be found on our investigation’s website: http://ouhscbbmc.github.io/MReportingPublic/.
The following graphs are not included in the final report.
10 Region Tables
The first table connects a county’s name to its Region ID, used in the previous graphs, as well as to its referral, enrollment, visits, and need. The second table displays each region’s random effects, estimated by the multi-level model.
Tables reflect the whole reporting period (which is 7 years long –from 2009-01-01 to 2015-12-31). Regions with at least one MIECHV county have a bolded ID. Counties not receiving C1 funding are indicated with an asterisk in the second column. More county and region information can be found on the online OSDH Maps document.
| ID | Counties | Referrals | Enrolls | Visits | Infants Born in Need |
|---|---|---|---|---|---|
| 1 | Blaine, Creek, Dewey*, Kingfisher, Lincoln, Logan | 956 | 932 | 18,635 | 7,784 |
| 2 | Beaver*, Cimarron*, Ellis*, Harper*, Texas, Woods*, Woodward | 185 | 179 | 2,974 | 3,485 |
| 3 | Kay, Noble*, Pawnee, Payne | 344 | 331 | 8,065 | 6,924 |
| 4 | Atoka, Coal, Pittsburg, Pontotoc | 377 | 375 | 6,734 | 5,207 |
| 5 | Nowata*, Osage*, Rogers, Washington | 369 | 363 | 7,293 | 5,877 |
| 6 | Bryan, Choctaw, McCurtain, Pushmataha* | 713 | 707 | 9,532 | 5,991 |
| 7 | Cleveland, McClain* | 657 | 645 | 20,349 | 7,884 |
| 8 | Latimer*, Le Flore | 282 | 278 | 7,539 | 3,504 |
| 9 | Garvin, Grady, Murray, Stephens | 321 | 292 | 5,511 | 6,206 |
| 10 | Canadian, Custer, Washita* | 355 | 349 | 8,034 | 5,301 |
| 11 | Adair, Muskogee, Sequoyah, Wagoner | 365 | 337 | 6,617 | 11,013 |
| 12 | Carter, Jefferson*, Johnston, Love*, Marshall | 461 | 442 | 8,312 | 5,036 |
| 13 | Haskell*, McIntosh, Okmulgee | 201 | 192 | 3,547 | 3,912 |
| 14 | Beckham, Greer*, Harmon*, Jackson, Roger Mills*, Tillman* | 255 | 255 | 4,064 | 4,187 |
| 15 | Alfalfa*, Garfield, Grant*, Major* | 378 | 362 | 7,008 | 3,992 |
| 16 | Hughes, Okfuskee*, Pottawatomie, Seminole | 452 | 418 | 7,460 | 6,423 |
| 17 | Caddo, Comanche, Cotton*, Kiowa* | 289 | 285 | 5,080 | 10,484 |
| 18 | Cherokee, Craig, Delaware, Mayes, Ottawa | 610 | 590 | 16,007 | 9,462 |
| 19 | Oklahoma | 1,804 | 1,761 | 35,178 | 44,163 |
| 20 | Tulsa | 1,999 | 1,991 | 53,491 | 27,704 |
| – | Entire State | 11,373 | 11,084 | 241,430 | 184,539 |
| ID | MIECHV | Referral Intercept |
Referral Cohort 1 |
Referral Cohort 2 |
Enroll Intercept |
Enroll Cohort 1 |
Enroll Cohort 2 |
Visit Intercept |
Visit Cohort 1 |
Visit Cohort 2 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | - | 0.61 | -0.24 | -0.40 | 0.46 | -0.21 | -0.57 | 0.34 | -0.02 | -0.13 |
| 2 | - | -0.01 | -0.23 | -0.41 | -0.02 | -0.17 | -0.44 | -0.04 | -0.14 | 0.08 |
| 3 | Yes | 0.21 | -0.26 | -0.23 | 0.18 | -0.25 | -0.52 | 0.13 | 0.01 | -0.17 |
| 4 | - | -0.22 | 0.07 | -0.57 | -0.25 | 0.07 | -0.62 | -0.39 | 0.01 | -0.07 |
| 5 | - | -0.12 | -0.10 | -0.51 | -0.03 | -0.15 | -0.97 | 0.22 | -0.08 | -0.16 |
| 6 | - | 0.24 | -0.15 | -1.04 | 0.26 | -0.16 | -1.16 | -0.36 | -0.07 | -0.18 |
| 7 | - | 0.22 | -0.01 | -0.58 | 0.24 | -0.04 | -0.97 | 0.63 | 0.09 | 0.08 |
| 8 | - | 0.20 | -0.19 | -0.42 | 0.22 | -0.23 | -0.67 | 0.59 | -0.06 | -0.05 |
| 9 | - | -0.44 | -0.10 | -0.08 | -0.38 | -0.15 | -0.33 | -0.43 | -0.09 | 0.01 |
| 10 | - | -0.11 | -0.13 | 0.10 | -0.15 | -0.11 | -0.01 | 0.42 | -0.01 | 0.07 |
| 11 | Yes | -0.20 | -0.13 | 0.07 | -0.24 | -0.17 | -0.19 | -0.61 | -0.06 | -0.02 |
| 12 | - | 0.27 | -0.06 | -0.52 | 0.31 | -0.07 | -1.02 | 0.47 | -0.01 | -0.26 |
| 13 | - | -0.40 | 0.27 | 0.07 | -0.38 | 0.24 | -0.23 | 0.15 | 0.06 | -0.10 |
| 14 | - | -0.11 | -0.12 | -0.45 | -0.05 | -0.15 | -0.84 | -0.38 | -0.07 | -0.22 |
| 15 | Yes | 1.06 | -0.38 | -0.46 | 1.07 | -0.40 | -1.09 | 0.97 | -0.01 | -0.10 |
| 16 | - | -0.11 | -0.21 | -0.13 | -0.08 | -0.23 | -0.70 | 0.37 | -0.07 | -0.14 |
| 17 | Yes | -0.52 | 0.29 | -0.32 | -0.51 | 0.27 | -0.43 | -0.68 | 0.06 | 0.17 |
| 18 | - | -0.32 | 0.08 | -0.02 | -0.33 | 0.09 | -0.22 | 0.16 | 0.06 | 0.04 |
| 19 | Yes | -0.49 | 0.15 | 0.30 | -0.53 | 0.15 | 0.17 | -1.56 | 0.10 | 0.13 |
| 20 | Yes | 0.24 | -0.15 | -0.31 | 0.20 | -0.15 | -0.36 | -0.01 | -0.04 | -0.05 |
11 Session Information
We would like to address any questions or suggestions during any stage of the evaluation. Please contact David Bard, Will Beasley, or Thomas Wilson in the BBMC (Biomedical and Behavioral Methodology Core) of OUHSC’s Pediatrics Department.
For the sake of documentation and reproducibility, the current report was rendered on a system using the following software.
Report rendered by Will at 2016-11-03, 14:28 -0500
Execution duration: 9.92018 minutes.R version 3.3.2 Patched (2016-10-31 r71611)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] lme4_1.1-12 Matrix_1.2-7.1 ggplot2_2.1.0 magrittr_1.5 knitr_1.14
loaded via a namespace (and not attached):
[1] Rcpp_0.12.7 splines_3.3.2 MASS_7.3-45 munsell_0.4.3 colorspace_1.2-7
[6] lattice_0.20-34 R6_2.2.0 minqa_1.2.4 highr_0.6 dplyr_0.5.0.9000
[11] stringr_1.1.0 plyr_1.8.4 tools_3.3.2 grid_3.3.2 gtable_0.2.0
[16] nlme_3.1-128 DBI_0.5-1 htmltools_0.3.5 lazyeval_0.2.0 yaml_2.1.13
[21] assertthat_0.1 digest_0.6.10 tibble_1.2 reshape2_1.4.2 tidyr_0.6.0
[26] nloptr_1.0.4 formatR_1.4 evaluate_0.10 rmarkdown_1.1 labeling_0.3
[31] stringi_1.1.2 scales_0.4.0 lubridate_1.6.0