How to calculate the 95% confidence interval from a model trained using CARET in R?

I have built different regression models using the R package caret. How can we calculate the 95% confidence interval for the predictions? I have followed the discussion noted here, however, it is not working.

rm(list = ls())
library(caret)

data("mtcars")
Train_data = mtcars[1:26, -c(8,9)]
Test_data = mtcars[27:32, -c(8,9)]


set.seed(100)
model_pls <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "pls", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_rf <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "ranger", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_svmr <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 8, 
  method = "svmRadial", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'),
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3,
  )
)

# This does not generate confidence interval
PLS.pred = predict(model_pls, subset(Test_data, select = -hp))  
RF.pred = predict(model_rf, subset(Test_data, select = -hp)) 
RF.svm = predict(model_svmr , subset(Test_data, select = -hp)) 


# This is not working
predict(model_pls$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_rf$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_svmr$finalModel, subset(Test_data, select = -hp), interval = "confidence")

Following Michael Matta's suggestion, I have tried the following code, however, it is not working as expected.

confint(model_pls, level = 0.95)
# Error in UseMethod("vcov"): no applicable method for 'vcov'

predict(model_pls, subset(Test_data, select = -hp), interval = "confidence")
# 64.47807  57.97479 151.59713 130.24356 183.20296  88.50035
# This does not show the CI.


Comments

Popular posts from this blog

Spring Elasticsearch Operations

Network Error and Timeout on Authorize.net JS

Object oriented programming concepts (OOPs)