You didn't define your x and y, but I will assume that they represent length and width.
So your first equation of xy = 30 is correct
Your second equation makes no sense
I will let x be one of the sides costing $55/ft
From the data, I get
Cost = 40x + 15(x+2y)
= 40x + 15x + 30y
but from xy=30 ----> y = 30/x
cost = 55x + 30(30/x) = 55x + 900/x
d(cost)/dx = 55 - 900/x^2
= 0 for a min of cost
55 = 900/x^2
x^2 = 900/55
x = 30/√55 = appr 4.05 ft
y = 30/x = appr 7.42 ft
check: (4.05)(7.42) = 30.051 , close enough
cost = 50(4.05) + 900/4.05) = $424.72
take a value of x slightly larger and smaller than 4.04
let x = 4
cost = 55(4) + 900/4 = $445 , which is more than my min
let x = 4.10
cost = 55(4.1) + 900/4.1 = $ 445.01 , again more than my min
So you have a pen about 4 ft by 7 ft to hold more than one billy goats?
Looks like in an effort to make the question sound "cute", the author of this question took no time to see if the resulting answer is even reasonable.
cost =