The Tigers were heavily beaten 58-16 by Hull Kingston Rovers last weekend, leaving Castleford 7th in the Super League table after 12 games.
The Tigers supporters were quick to let their feelings known following the full time hooter last weekend, but Milner says it is now up to the team to improve their performance against St Helens.
“It was embarrassing, an absolutely abysmal performance from us and we’ve got to put the wrongs right on Sunday,” Milner told the Pontefract and Castleford Express.
“We owe it to the fans, the coaching staff and ourselves – it was just appalling. We’ve had some pretty tough words said about each other since and we’re looking to respond. It’s about moving forward and getting back to winning ways.
“The fans absolutely let us know about it and they had every right to do so. We owe them a good performance now. To play like that at home is not acceptable.
“Myself and the rest of the boys are all ashamed of that performance. It will be tough against St Helens, they got a good win last week, but we need to really knuckle down and get these performances improving and get the two points.”