RTOS Kernel performance:
Traditionally measured by measuring the syscall lenghts
(real-time executive assumption: non-preemptable syscalls - maybe with
interrupts disabled - and irrelevant IRQ load).
First scientific paper introducing the "kernel latency" concept,
and using it to systematically measure the real-time OS performance:
Luca Abeni, Ashvin Goel, Charles Krasic, Jim Snow, and Jonathan Walpole. "A measurement-based analysis of the real-time performance of linux" Real-Time and Embedded Technology and Applications Symposium, 2002. Proceedings. Eighth IEEE. IEEE, 2002.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.7.3497&rep=rep1&type=pdf



RTLinux:

Michael Barabanov and Victor Yodaiken. "A Real-Time Linux" Linux Journal (1996)
http://www.yodaiken.com/papers/ARealTimeLinux.pdf

The "Adeos nanokernel" (and the i-pipe architecture) implement concepts
described in the following papers:

D. Probert, J. Bruno, and M. Karzaorman. "Space: a new approach to operating system abstraction." Proceedings of the International Workshop on Object Orientation in Operating Systems, pages 133-137, October 1991.

D. Probert, J. Bruno. "Building fundamentally extensible application-specific operating systems in Space", March 1995.

D. Cheriton, K. Duda. "A caching model of operating system kernel functionality". Proceedings of the Symposium on Operating Systems Design and Implementation, pages 179-194, Monterey CA (USA), 1994.

D. Engler, M. Kaashoek, and J. O'Toole Jr. "Exokernel: an operating system architecture for application-specific resource management", December 1995.

All of them have been published before RTLinux came to life.
(although the basic concepts were known years before, the first official
reference I found is a paper from 1991).

Some concepts similar to the ones implemented in Adeos / i-pipe have been
implemented in the Xtratum nanokernel (sometimes known as Xtratum hypervisor):
http://www.xtratum.org/
The main difference between Xtratum and Adeos is that Adeos is based on
Linux (if we want to see it as an hypervisor, we can say that it is a
hosted hypervisor), while Xtratum is independent from Linux (can be seen
as a bare metal hypervisor). 



Microkernels and Real-Time:

Jochen Liedtke. "μ-kernels must and can be small" Proceedings of the Fifth International Workshop on Object-Orientation in Operation Systems (IWOOOS), Seattle, WA, 1996, pp. 152-155.
doi: 10.1109/IWOOOS.1996.557909
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.2355&rep=rep1&type=pdf

Joken Liedtke. "On μ-Kernel Construction." Proceedings of the 15th ACM Symposium on Operating System Principles (SOSP). 1995.
http://www.tudos.de/papers_ps/jochen/Mikern.ps

Hermann Härtig, Michael Hohmuth, and Jean Wolter. "Taming linux" Proceedings of the 5th Annual Australasian Conference on Parallel And Real-Time Systems (PART’98). 1998.
https://os.inf.tu-dresden.de/papers_ps/part98.pdf

Michael Hohmuth and Hermann Härtig. "Pragmatic Nonblocking Synchronization for Real-Time Systems" USENIX Annual Technical Conference, General Track. 2001.
https://os.inf.tu-dresden.de/papers_ps/usenix2001.pdf

Frank Mehnert, Michael Hohmuth, and Hermann Hartig. "Cost and benefit of separate address spaces in real-time operating systems" Real-Time Systems Symposium, 2002. RTSS 2002. 23rd IEEE. IEEE, 2002.
https://os.inf.tu-dresden.de/papers_ps/rtss2002.pdf