To validate transactions, Bitcoin uses Elliptic Curve cryptography. Specifically, an Elliptic Curve Digital Signature Algorithm (ECDSA) on the curve secp256k1.
ECDSA includes three important pieces of data. The private key (a large random number), the public key and a digital signature. These pieces of data have a mathematical relationship such that the public key and a digital signature can inexpensively be computed from the private key, but the private key cannot be computed inexpensively from either the public key or a single digital signature.
There also exists a mathematical relationship between the public key and a digital signature which allows anyone to inexpensively verify that a digital signature was computed from the same private key used to compute the public key. Knowledge of the private key is not required to know that a digital signature was computed from the same private key as the public key.
How can we know that a digital signature is authentic without knowledge of the private key?
By knowledge of the Elliptic Curve parameters and the four following equations...
(1) Qa = dA * G
(2) P = k * G
(3) S = k^-1 (z + dA * R) mod p
(4) P = S^-1 * z * G + S^-1 * R * Qa
Qa is the public key,
dA is the private key,
G is the point of reference in our curve parameters,
P is a point on our Elliptic Curve,
k is a large random number (not the private key),
S is the second half of the digital signature,
z is the hash of the message being signed (encoded as a large integer),
R is the first half of the digital signature (and also the x coordinate of P),
p is a very large prime number defined by our curve parameters.
And then, by substituting (1) into (4) we have...
(5) P = S^-1 * z * G + S^-1 * R * dA * G
Factorizing (5) leaves us with...
(6) P = S^-1 (z + R * dA) * G
Now, substituting in (2), we have...
(7) k * G = S^-1 (z + R * dA) * G
Which can be simplified to...
(8) k = S^-1 (z + R * dA)
Lastly, we invert k and S leaving us with...
(9) S = k^-1 (z + R * dA)
Which is the equation first used to create the signature by the private key holder.
This proof informs us that if the x coordinate of P from (4) is equal to R, then the digital signature (R, S) was generated from the private key dA.
Please see here for a longer explanation and here and here to understand how ECDSA is applied in the Bitcoin system.