All the students in a school are made to stand in rows so as to form an equilateral triangle. The first row consists of one student, the second row 2 students, third row 3 students and so on. Had there been 90 more students, the total number of students could have been arranged in the shape of a square, so that each side of the square has 5 students less than the number of students in the side of the triangle. Find the total number of students initially present in the school?

Answer 1

You are dealing with the triangular numbers whose sums of consecutive rows would be

1 3 6 10 15 21 28 ...

these numbers are generated by (n^2+n)/2 for n equal to the set of natural numbers.

So I added 90 to each of these to see if I get a perfect square.
The first such n value was at n=4
the sum of students would have been 10, and adding 90 gave me 100.
But that did not satisfy the second condition you gave, namely that the side of the square had to be 5 less than the side of the triangle.

I kept going for a while, almost gave up, but came to n = 26

the sum of the students in the triangle is (26^2 + 26)/2 = 351
adding 90 got me 441 which is 21^2
and the difference between these is indeed 5

so there were initially 351 students in the school

Answer 2

as first row consist 1 student, 2nd row consist 2 students and so on...
let the no of rows in triangle be "n" now the total no of students in triangle will be 1+2+3+....+n=n(n+1)/2
given if 90 more students are added that is n(n+1)/2+90 is equal to no of students in square arranged in such a manner that has 5 less students each side than the triangle.
so now n(n+1)/2+90=(n-5)^2 (area of the square )
n^2-21n-130=0
n=26 substituting in n(n+1)/2 we get 351.
answer is 351