PGM.cpp@ 26112

最後變更在這個檔案從26112是 26107,由 vboxsync 提交於 15 年前
PGM: Some harmless page counting error and factored out the pure MMIO pages from the zero page count. (should probably drop all MMIO pages out of the zero page count later)
屬性 svn:eol-style 設為 `native` 屬性 svn:keywords 設為 `Id`
檔案大小: 216.0 KB

行
1	/* $Id: PGM.cpp 26107 2010-01-30 14:18:03Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2007 Sun Microsystems, Inc.
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.alldomusa.eu.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*
17	* Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa
18	* Clara, CA 95054 USA or visit http://www.sun.com if you need
19	* additional information or have any questions.
20	*/
21
22
23	/** @page pg_pgm PGM - The Page Manager and Monitor
24	*
25	* @see grp_pgm,
26	* @ref pg_pgm_pool,
27	* @ref pg_pgm_phys.
28	*
29	*
30	* @section sec_pgm_modes Paging Modes
31	*
32	* There are three memory contexts: Host Context (HC), Guest Context (GC)
33	* and intermediate context. When talking about paging HC can also be refered to
34	* as "host paging", and GC refered to as "shadow paging".
35	*
36	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
37	* is defined by the host operating system. The mode used in the shadow paging mode
38	* depends on the host paging mode and what the mode the guest is currently in. The
39	* following relation between the two is defined:
40	*
41	* @verbatim
42	Host > 32-bit \| PAE \| AMD64 \|
43	Guest \| \| \| \|
44	==v================================
45	32-bit 32-bit PAE PAE
46	-------\|--------\|--------\|--------\|
47	PAE PAE PAE PAE
48	-------\|--------\|--------\|--------\|
49	AMD64 AMD64 AMD64 AMD64
50	-------\|--------\|--------\|--------\| @endverbatim
51	*
52	* All configuration except those in the diagonal (upper left) are expected to
53	* require special effort from the switcher (i.e. a bit slower).
54	*
55	*
56	*
57	*
58	* @section sec_pgm_shw The Shadow Memory Context
59	*
60	*
61	* [..]
62	*
63	* Because of guest context mappings requires PDPT and PML4 entries to allow
64	* writing on AMD64, the two upper levels will have fixed flags whatever the
65	* guest is thinking of using there. So, when shadowing the PD level we will
66	* calculate the effective flags of PD and all the higher levels. In legacy
67	* PAE mode this only applies to the PWT and PCD bits (the rest are
68	* ignored/reserved/MBZ). We will ignore those bits for the present.
69	*
70	*
71	*
72	* @section sec_pgm_int The Intermediate Memory Context
73	*
74	* The world switch goes thru an intermediate memory context which purpose it is
75	* to provide different mappings of the switcher code. All guest mappings are also
76	* present in this context.
77	*
78	* The switcher code is mapped at the same location as on the host, at an
79	* identity mapped location (physical equals virtual address), and at the
80	* hypervisor location. The identity mapped location is for when the world
81	* switches that involves disabling paging.
82	*
83	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
84	* simplifies switching guest CPU mode and consistency at the cost of more
85	* code to do the work. All memory use for those page tables is located below
86	* 4GB (this includes page tables for guest context mappings).
87	*
88	*
89	* @subsection subsec_pgm_int_gc Guest Context Mappings
90	*
91	* During assignment and relocation of a guest context mapping the intermediate
92	* memory context is used to verify the new location.
93	*
94	* Guest context mappings are currently restricted to below 4GB, for reasons
95	* of simplicity. This may change when we implement AMD64 support.
96	*
97	*
98	*
99	*
100	* @section sec_pgm_misc Misc
101	*
102	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
103	*
104	* The differences between legacy PAE and long mode PAE are:
105	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
106	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
107	* usual meanings while 6 is ignored (AMD). This means that upon switching to
108	* legacy PAE mode we'll have to clear these bits and when going to long mode
109	* they must be set. This applies to both intermediate and shadow contexts,
110	* however we don't need to do it for the intermediate one since we're
111	* executing with CR0.WP at that time.
112	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
113	* a page aligned one is required.
114	*
115	*
116	* @section sec_pgm_handlers Access Handlers
117	*
118	* Placeholder.
119	*
120	*
121	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
122	*
123	* Placeholder.
124	*
125	*
126	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
127	*
128	* We currently implement three types of virtual access handlers: ALL, WRITE
129	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
130	*
131	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
132	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
133	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
134	* rest of this section is going to be about these handlers.
135	*
136	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
137	* how successfull this is gonna be...
138	*
139	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
140	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
141	* and create a new node that is inserted into the AVL tree (range key). Then
142	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
143	*
144	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
145	*
146	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
147	* via the current guest CR3 and update the physical page -> virtual handler
148	* translation. Needless to say, this doesn't exactly scale very well. If any changes
149	* are detected, it will flag a virtual bit update just like we did on registration.
150	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
151	*
152	* 2b. The virtual bit update process will iterate all the pages covered by all the
153	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
154	* virtual handlers on that page.
155	*
156	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
157	* we don't miss any alias mappings of the monitored pages.
158	*
159	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
160	*
161	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
162	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
163	* will call the handlers like in the next step. If the physical mapping has
164	* changed we will - some time in the future - perform a handler callback
165	* (optional) and update the physical -> virtual handler cache.
166	*
167	* 4. \#PF(,write) on a page in the range. This will cause the handler to
168	* be invoked.
169	*
170	* 5. The guest invalidates the page and changes the physical backing or
171	* unmaps it. This should cause the invalidation callback to be invoked
172	* (it might not yet be 100% perfect). Exactly what happens next... is
173	* this where we mess up and end up out of sync for a while?
174	*
175	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
176	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
177	* this handler to NONE and trigger a full PGM resync (basically the same
178	* as int step 1). Which means 2 is executed again.
179	*
180	*
181	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
182	*
183	* There is a bunch of things that needs to be done to make the virtual handlers
184	* work 100% correctly and work more efficiently.
185	*
186	* The first bit hasn't been implemented yet because it's going to slow the
187	* whole mess down even more, and besides it seems to be working reliably for
188	* our current uses. OTOH, some of the optimizations might end up more or less
189	* implementing the missing bits, so we'll see.
190	*
191	* On the optimization side, the first thing to do is to try avoid unnecessary
192	* cache flushing. Then try team up with the shadowing code to track changes
193	* in mappings by means of access to them (shadow in), updates to shadows pages,
194	* invlpg, and shadow PT discarding (perhaps).
195	*
196	* Some idea that have popped up for optimization for current and new features:
197	* - bitmap indicating where there are virtual handlers installed.
198	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
199	* - Further optimize this by min/max (needs min/max avl getters).
200	* - Shadow page table entry bit (if any left)?
201	*
202	*/
203
204
205	/** @page pg_pgm_phys PGM Physical Guest Memory Management
206	*
207	*
208	* Objectives:
209	* - Guest RAM over-commitment using memory ballooning,
210	* zero pages and general page sharing.
211	* - Moving or mirroring a VM onto a different physical machine.
212	*
213	*
214	* @subsection subsec_pgmPhys_Definitions Definitions
215	*
216	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
217	* machinery assoicated with it.
218	*
219	*
220	*
221	*
222	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
223	*
224	* Initially we map all guest memory to the (per VM) zero page, which
225	* means that none of the read functions will cause pages to be allocated.
226	*
227	* Exception, access bit in page tables that have been shared. This must
228	* be handled, but we must also make sure PGMGst*Modify doesn't make
229	* unnecessary modifications.
230	*
231	* Allocation points:
232	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
233	* - Replacing a zero page mapping at \#PF.
234	* - Replacing a shared page mapping at \#PF.
235	* - ROM registration (currently MMR3RomRegister).
236	* - VM restore (pgmR3Load).
237	*
238	* For the first three it would make sense to keep a few pages handy
239	* until we've reached the max memory commitment for the VM.
240	*
241	* For the ROM registration, we know exactly how many pages we need
242	* and will request these from ring-0. For restore, we will save
243	* the number of non-zero pages in the saved state and allocate
244	* them up front. This would allow the ring-0 component to refuse
245	* the request if the isn't sufficient memory available for VM use.
246	*
247	* Btw. for both ROM and restore allocations we won't be requiring
248	* zeroed pages as they are going to be filled instantly.
249	*
250	*
251	* @subsection subsec_pgmPhys_FreePage Freeing a page
252	*
253	* There are a few points where a page can be freed:
254	* - After being replaced by the zero page.
255	* - After being replaced by a shared page.
256	* - After being ballooned by the guest additions.
257	* - At reset.
258	* - At restore.
259	*
260	* When freeing one or more pages they will be returned to the ring-0
261	* component and replaced by the zero page.
262	*
263	* The reasoning for clearing out all the pages on reset is that it will
264	* return us to the exact same state as on power on, and may thereby help
265	* us reduce the memory load on the system. Further it might have a
266	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
267	*
268	* On restore, as mention under the allocation topic, pages should be
269	* freed / allocated depending on how many is actually required by the
270	* new VM state. The simplest approach is to do like on reset, and free
271	* all non-ROM pages and then allocate what we need.
272	*
273	* A measure to prevent some fragmentation, would be to let each allocation
274	* chunk have some affinity towards the VM having allocated the most pages
275	* from it. Also, try make sure to allocate from allocation chunks that
276	* are almost full. Admittedly, both these measures might work counter to
277	* our intentions and its probably not worth putting a lot of effort,
278	* cpu time or memory into this.
279	*
280	*
281	* @subsection subsec_pgmPhys_SharePage Sharing a page
282	*
283	* The basic idea is that there there will be a idle priority kernel
284	* thread walking the non-shared VM pages hashing them and looking for
285	* pages with the same checksum. If such pages are found, it will compare
286	* them byte-by-byte to see if they actually are identical. If found to be
287	* identical it will allocate a shared page, copy the content, check that
288	* the page didn't change while doing this, and finally request both the
289	* VMs to use the shared page instead. If the page is all zeros (special
290	* checksum and byte-by-byte check) it will request the VM that owns it
291	* to replace it with the zero page.
292	*
293	* To make this efficient, we will have to make sure not to try share a page
294	* that will change its contents soon. This part requires the most work.
295	* A simple idea would be to request the VM to write monitor the page for
296	* a while to make sure it isn't modified any time soon. Also, it may
297	* make sense to skip pages that are being write monitored since this
298	* information is readily available to the thread if it works on the
299	* per-VM guest memory structures (presently called PGMRAMRANGE).
300	*
301	*
302	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
303	*
304	* The pages are organized in allocation chunks in ring-0, this is a necessity
305	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
306	* could easily work on a page-by-page basis if we liked. Whether this is possible
307	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
308	* become a problem as part of the idea here is that we wish to return memory to
309	* the host system.
310	*
311	* For instance, starting two VMs at the same time, they will both allocate the
312	* guest memory on-demand and if permitted their page allocations will be
313	* intermixed. Shut down one of the two VMs and it will be difficult to return
314	* any memory to the host system because the page allocation for the two VMs are
315	* mixed up in the same allocation chunks.
316	*
317	* To further complicate matters, when pages are freed because they have been
318	* ballooned or become shared/zero the whole idea is that the page is supposed
319	* to be reused by another VM or returned to the host system. This will cause
320	* allocation chunks to contain pages belonging to different VMs and prevent
321	* returning memory to the host when one of those VM shuts down.
322	*
323	* The only way to really deal with this problem is to move pages. This can
324	* either be done at VM shutdown and or by the idle priority worker thread
325	* that will be responsible for finding sharable/zero pages. The mechanisms
326	* involved for coercing a VM to move a page (or to do it for it) will be
327	* the same as when telling it to share/zero a page.
328	*
329	*
330	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
331	*
332	* There's a difficult balance between keeping the per-page tracking structures
333	* (global and guest page) easy to use and keeping them from eating too much
334	* memory. We have limited virtual memory resources available when operating in
335	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
336	* tracking structures will be attemted designed such that we can deal with up
337	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
338	*
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
341	*
342	* @see pg_GMM
343	*
344	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
345	*
346	* Fixed info is the physical address of the page (HCPhys) and the page id
347	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
348	* Today we've restricting ourselves to 40(-12) bits because this is the current
349	* restrictions of all AMD64 implementations (I think Barcelona will up this
350	* to 48(-12) bits, not that it really matters) and I needed the bits for
351	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
352	* decent range for the page id: 2^(28+12) = 1024TB.
353	*
354	* In additions to these, we'll have to keep maintaining the page flags as we
355	* currently do. Although it wouldn't harm to optimize these quite a bit, like
356	* for instance the ROM shouldn't depend on having a write handler installed
357	* in order for it to become read-only. A RO/RW bit should be considered so
358	* that the page syncing code doesn't have to mess about checking multiple
359	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
360	* figure out how to setup a shadow PTE. But this of course, is second
361	* priority at present. Current this requires 12 bits, but could probably
362	* be optimized to ~8.
363	*
364	* Then there's the 24 bits used to track which shadow page tables are
365	* currently mapping a page for the purpose of speeding up physical
366	* access handlers, and thereby the page pool cache. More bit for this
367	* purpose wouldn't hurt IIRC.
368	*
369	* Then there is a new bit in which we need to record what kind of page
370	* this is, shared, zero, normal or write-monitored-normal. This'll
371	* require 2 bits. One bit might be needed for indicating whether a
372	* write monitored page has been written to. And yet another one or
373	* two for tracking migration status. 3-4 bits total then.
374	*
375	* Whatever is left will can be used to record the sharabilitiy of a
376	* page. The page checksum will not be stored in the per-VM table as
377	* the idle thread will not be permitted to do modifications to it.
378	* It will instead have to keep its own working set of potentially
379	* shareable pages and their check sums and stuff.
380	*
381	* For the present we'll keep the current packing of the
382	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
383	* we'll have to change it to a struct with a total of 128-bits at
384	* our disposal.
385	*
386	* The initial layout will be like this:
387	* @verbatim
388	RTHCPHYS HCPhys; The current stuff.
389	63:40 Current shadow PT tracking stuff.
390	39:12 The physical page frame number.
391	11:0 The current flags.
392	uint32_t u28PageId : 28; The page id.
393	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
394	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
395	uint32_t u1Reserved : 1; Reserved for later.
396	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
397	@endverbatim
398	*
399	* The final layout will be something like this:
400	* @verbatim
401	RTHCPHYS HCPhys; The current stuff.
402	63:48 High page id (12+).
403	47:12 The physical page frame number.
404	11:0 Low page id.
405	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
406	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
407	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
408	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
409	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
410	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
411	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
412	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
413	@endverbatim
414	*
415	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
416	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
417	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
418	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
419	*
420	* A couple of cost examples for the total cost per-VM + kernel.
421	* 32-bit Windows and 32-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
423	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
424	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
425	* 64-bit Windows and 64-bit linux:
426	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
427	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
428	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
429	*
430	* UPDATE - 2007-09-27:
431	* Will need a ballooned flag/state too because we cannot
432	* trust the guest 100% and reporting the same page as ballooned more
433	* than once will put the GMM off balance.
434	*
435	*
436	* @subsection subsec_pgmPhys_Serializing Serializing Access
437	*
438	* Initially, we'll try a simple scheme:
439	*
440	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
441	* by the EMT thread of that VM while in the pgm critsect.
442	* - Other threads in the VM process that needs to make reliable use of
443	* the per-VM RAM tracking structures will enter the critsect.
444	* - No process external thread or kernel thread will ever try enter
445	* the pgm critical section, as that just won't work.
446	* - The idle thread (and similar threads) doesn't not need 100% reliable
447	* data when performing it tasks as the EMT thread will be the one to
448	* do the actual changes later anyway. So, as long as it only accesses
449	* the main ram range, it can do so by somehow preventing the VM from
450	* being destroyed while it works on it...
451	*
452	* - The over-commitment management, including the allocating/freeing
453	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
454	* more mundane mutex implementation is broken on Linux).
455	* - A separeate mutex is protecting the set of allocation chunks so
456	* that pages can be shared or/and freed up while some other VM is
457	* allocating more chunks. This mutex can be take from under the other
458	* one, but not the otherway around.
459	*
460	*
461	* @subsection subsec_pgmPhys_Request VM Request interface
462	*
463	* When in ring-0 it will become necessary to send requests to a VM so it can
464	* for instance move a page while defragmenting during VM destroy. The idle
465	* thread will make use of this interface to request VMs to setup shared
466	* pages and to perform write monitoring of pages.
467	*
468	* I would propose an interface similar to the current VMReq interface, similar
469	* in that it doesn't require locking and that the one sending the request may
470	* wait for completion if it wishes to. This shouldn't be very difficult to
471	* realize.
472	*
473	* The requests themselves are also pretty simple. They are basically:
474	* -# Check that some precondition is still true.
475	* -# Do the update.
476	* -# Update all shadow page tables involved with the page.
477	*
478	* The 3rd step is identical to what we're already doing when updating a
479	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
480	*
481	*
482	*
483	* @section sec_pgmPhys_MappingCaches Mapping Caches
484	*
485	* In order to be able to map in and out memory and to be able to support
486	* guest with more RAM than we've got virtual address space, we'll employing
487	* a mapping cache. There is already a tiny one for GC (see PGMGCDynMapGCPageEx)
488	* and we'll create a similar one for ring-0 unless we decide to setup a dedicate
489	* memory context for the HWACCM execution.
490	*
491	*
492	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
493	*
494	* We've considered implementing the ring-3 mapping cache page based but found
495	* that this was bother some when one had to take into account TLBs+SMP and
496	* portability (missing the necessary APIs on several platforms). There were
497	* also some performance concerns with this approach which hadn't quite been
498	* worked out.
499	*
500	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
501	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
502	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
503	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
504	* costly than a single page, although how much more costly is uncertain. We'll
505	* try address this by using a very big cache, preferably bigger than the actual
506	* VM RAM size if possible. The current VM RAM sizes should give some idea for
507	* 32-bit boxes, while on 64-bit we can probably get away with employing an
508	* unlimited cache.
509	*
510	* The cache have to parts, as already indicated, the ring-3 side and the
511	* ring-0 side.
512	*
513	* The ring-0 will be tied to the page allocator since it will operate on the
514	* memory objects it contains. It will therefore require the first ring-0 mutex
515	* discussed in @ref subsec_pgmPhys_Serializing. We
516	* some double house keeping wrt to who has mapped what I think, since both
517	* VMMR0.r0 and RTR0MemObj will keep track of mapping relataions
518	*
519	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
520	* require anyone that desires to do changes to the mapping cache to do that
521	* from within this critsect. Alternatively, we could employ a separate critsect
522	* for serializing changes to the mapping cache as this would reduce potential
523	* contention with other threads accessing mappings unrelated to the changes
524	* that are in process. We can see about this later, contention will show
525	* up in the statistics anyway, so it'll be simple to tell.
526	*
527	* The organization of the ring-3 part will be very much like how the allocation
528	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
529	* having to walk the tree all the time, we'll have a couple of lookaside entries
530	* like in we do for I/O ports and MMIO in IOM.
531	*
532	* The simplified flow of a PGMPhysRead/Write function:
533	* -# Enter the PGM critsect.
534	* -# Lookup GCPhys in the ram ranges and get the Page ID.
535	* -# Calc the Allocation Chunk ID from the Page ID.
536	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
537	* If not found in cache:
538	* -# Call ring-0 and request it to be mapped and supply
539	* a chunk to be unmapped if the cache is maxed out already.
540	* -# Insert the new mapping into the AVL tree (id + R3 address).
541	* -# Update the relevant lookaside entry and return the mapping address.
542	* -# Do the read/write according to monitoring flags and everything.
543	* -# Leave the critsect.
544	*
545	*
546	* @section sec_pgmPhys_Fallback Fallback
547	*
548	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
549	* API and thus require a fallback.
550	*
551	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
552	* will return to the ring-3 caller (and later ring-0) and asking it to seed
553	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
554	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
555	* "SeededAllocPages" call to ring-0.
556	*
557	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
558	* all page sharing (zero page detection will continue). It will also force
559	* all allocations to come from the VM which seeded the page. Both these
560	* measures are taken to make sure that there will never be any need for
561	* mapping anything into ring-3 - everything will be mapped already.
562	*
563	* Whether we'll continue to use the current MM locked memory management
564	* for this I don't quite know (I'd prefer not to and just ditch that all
565	* togther), we'll see what's simplest to do.
566	*
567	*
568	*
569	* @section sec_pgmPhys_Changes Changes
570	*
571	* Breakdown of the changes involved?
572	*/
573
574	/*******************************************************************************
575	* Header Files *
576	*******************************************************************************/
577	#define LOG_GROUP LOG_GROUP_PGM
578	#include <VBox/dbgf.h>
579	#include <VBox/pgm.h>
580	#include <VBox/cpum.h>
581	#include <VBox/iom.h>
582	#include <VBox/sup.h>
583	#include <VBox/mm.h>
584	#include <VBox/em.h>
585	#include <VBox/stam.h>
586	#include <VBox/rem.h>
587	#include <VBox/selm.h>
588	#include <VBox/ssm.h>
589	#include <VBox/hwaccm.h>
590	#include "PGMInternal.h"
591	#include <VBox/vm.h>
592
593	#include <VBox/dbg.h>
594	#include <VBox/param.h>
595	#include <VBox/err.h>
596
597	#include <iprt/asm.h>
598	#include <iprt/assert.h>
599	#include <iprt/env.h>
600	#include <iprt/mem.h>
601	#include <iprt/file.h>
602	#include <iprt/string.h>
603	#include <iprt/thread.h>
604
605
606	/*******************************************************************************
607	* Defined Constants And Macros *
608	*******************************************************************************/
609	/** Saved state data unit version for 2.5.x and later. */
610	#define PGM_SAVED_STATE_VERSION 9
611	/** Saved state data unit version for 2.2.2 and later. */
612	#define PGM_SAVED_STATE_VERSION_2_2_2 8
613	/** Saved state data unit version for 2.2.0. */
614	#define PGM_SAVED_STATE_VERSION_RR_DESC 7
615	/** Saved state data unit version. */
616	#define PGM_SAVED_STATE_VERSION_OLD_PHYS_CODE 6
617
618
619	/*******************************************************************************
620	* Internal Functions *
621	*******************************************************************************/
622	static int pgmR3InitPaging(PVM pVM);
623	static void pgmR3InitStats(PVM pVM);
624	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
625	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
626	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
627	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
628	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
629	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
630	#ifdef VBOX_STRICT
631	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
632	#endif
633	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
634	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
635	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
636
637	#ifdef VBOX_WITH_DEBUGGER
638	/** @todo Convert the first two commands to 'info' items. */
639	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
640	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
641	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
642	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
643	# ifdef VBOX_STRICT
644	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
645	# endif
646	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
647	#endif
648
649
650	/*******************************************************************************
651	* Global Variables *
652	*******************************************************************************/
653	#ifdef VBOX_WITH_DEBUGGER
654	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
655	static const DBGCVARDESC g_aPgmErrorArgs[] =
656	{
657	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
658	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
659	};
660
661	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
662	{
663	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
664	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
665	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
666	};
667
668	/** Command descriptors. */
669	static const DBGCCMD g_aCmds[] =
670	{
671	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, pResultDesc, fFlags, pfnHandler pszSyntax, ....pszDescription */
672	{ "pgmram", 0, 0, NULL, 0, NULL, 0, pgmR3CmdRam, "", "Display the ram ranges." },
673	{ "pgmsync", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
674	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, NULL, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
675	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, NULL, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
676	#ifdef VBOX_STRICT
677	{ "pgmassertcr3", 0, 0, NULL, 0, NULL, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
678	#endif
679	{ "pgmsyncalways", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
680	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, NULL, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
681	};
682	#endif
683
684
685
686
687	/*
688	* Shadow - 32-bit mode
689	*/
690	#define PGM_SHW_TYPE PGM_TYPE_32BIT
691	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
692	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
693	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
694	#include "PGMShw.h"
695
696	/* Guest - real mode */
697	#define PGM_GST_TYPE PGM_TYPE_REAL
698	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
699	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
700	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
701	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
702	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
703	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
704	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
705	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
706	#include "PGMBth.h"
707	#include "PGMGstDefs.h"
708	#include "PGMGst.h"
709	#undef BTH_PGMPOOLKIND_PT_FOR_PT
710	#undef BTH_PGMPOOLKIND_ROOT
711	#undef PGM_BTH_NAME
712	#undef PGM_BTH_NAME_RC_STR
713	#undef PGM_BTH_NAME_R0_STR
714	#undef PGM_GST_TYPE
715	#undef PGM_GST_NAME
716	#undef PGM_GST_NAME_RC_STR
717	#undef PGM_GST_NAME_R0_STR
718
719	/* Guest - protected mode */
720	#define PGM_GST_TYPE PGM_TYPE_PROT
721	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
722	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
723	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
724	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
725	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
726	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
727	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
728	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
729	#include "PGMBth.h"
730	#include "PGMGstDefs.h"
731	#include "PGMGst.h"
732	#undef BTH_PGMPOOLKIND_PT_FOR_PT
733	#undef BTH_PGMPOOLKIND_ROOT
734	#undef PGM_BTH_NAME
735	#undef PGM_BTH_NAME_RC_STR
736	#undef PGM_BTH_NAME_R0_STR
737	#undef PGM_GST_TYPE
738	#undef PGM_GST_NAME
739	#undef PGM_GST_NAME_RC_STR
740	#undef PGM_GST_NAME_R0_STR
741
742	/* Guest - 32-bit mode */
743	#define PGM_GST_TYPE PGM_TYPE_32BIT
744	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
745	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
746	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
747	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
748	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
749	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
750	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
751	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
752	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
753	#include "PGMBth.h"
754	#include "PGMGstDefs.h"
755	#include "PGMGst.h"
756	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
757	#undef BTH_PGMPOOLKIND_PT_FOR_PT
758	#undef BTH_PGMPOOLKIND_ROOT
759	#undef PGM_BTH_NAME
760	#undef PGM_BTH_NAME_RC_STR
761	#undef PGM_BTH_NAME_R0_STR
762	#undef PGM_GST_TYPE
763	#undef PGM_GST_NAME
764	#undef PGM_GST_NAME_RC_STR
765	#undef PGM_GST_NAME_R0_STR
766
767	#undef PGM_SHW_TYPE
768	#undef PGM_SHW_NAME
769	#undef PGM_SHW_NAME_RC_STR
770	#undef PGM_SHW_NAME_R0_STR
771
772
773	/*
774	* Shadow - PAE mode
775	*/
776	#define PGM_SHW_TYPE PGM_TYPE_PAE
777	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
778	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
779	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
780	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
781	#include "PGMShw.h"
782
783	/* Guest - real mode */
784	#define PGM_GST_TYPE PGM_TYPE_REAL
785	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
792	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
793	#include "PGMGstDefs.h"
794	#include "PGMBth.h"
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef BTH_PGMPOOLKIND_ROOT
797	#undef PGM_BTH_NAME
798	#undef PGM_BTH_NAME_RC_STR
799	#undef PGM_BTH_NAME_R0_STR
800	#undef PGM_GST_TYPE
801	#undef PGM_GST_NAME
802	#undef PGM_GST_NAME_RC_STR
803	#undef PGM_GST_NAME_R0_STR
804
805	/* Guest - protected mode */
806	#define PGM_GST_TYPE PGM_TYPE_PROT
807	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
808	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
809	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
810	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
811	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
812	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
813	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
814	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
815	#include "PGMGstDefs.h"
816	#include "PGMBth.h"
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef BTH_PGMPOOLKIND_ROOT
819	#undef PGM_BTH_NAME
820	#undef PGM_BTH_NAME_RC_STR
821	#undef PGM_BTH_NAME_R0_STR
822	#undef PGM_GST_TYPE
823	#undef PGM_GST_NAME
824	#undef PGM_GST_NAME_RC_STR
825	#undef PGM_GST_NAME_R0_STR
826
827	/* Guest - 32-bit mode */
828	#define PGM_GST_TYPE PGM_TYPE_32BIT
829	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
830	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
831	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
832	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
833	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
834	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
835	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
836	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
837	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
838	#include "PGMGstDefs.h"
839	#include "PGMBth.h"
840	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
841	#undef BTH_PGMPOOLKIND_PT_FOR_PT
842	#undef BTH_PGMPOOLKIND_ROOT
843	#undef PGM_BTH_NAME
844	#undef PGM_BTH_NAME_RC_STR
845	#undef PGM_BTH_NAME_R0_STR
846	#undef PGM_GST_TYPE
847	#undef PGM_GST_NAME
848	#undef PGM_GST_NAME_RC_STR
849	#undef PGM_GST_NAME_R0_STR
850
851	/* Guest - PAE mode */
852	#define PGM_GST_TYPE PGM_TYPE_PAE
853	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
854	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
855	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
856	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
857	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
858	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
859	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
860	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
861	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
862	#include "PGMBth.h"
863	#include "PGMGstDefs.h"
864	#include "PGMGst.h"
865	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
866	#undef BTH_PGMPOOLKIND_PT_FOR_PT
867	#undef BTH_PGMPOOLKIND_ROOT
868	#undef PGM_BTH_NAME
869	#undef PGM_BTH_NAME_RC_STR
870	#undef PGM_BTH_NAME_R0_STR
871	#undef PGM_GST_TYPE
872	#undef PGM_GST_NAME
873	#undef PGM_GST_NAME_RC_STR
874	#undef PGM_GST_NAME_R0_STR
875
876	#undef PGM_SHW_TYPE
877	#undef PGM_SHW_NAME
878	#undef PGM_SHW_NAME_RC_STR
879	#undef PGM_SHW_NAME_R0_STR
880
881
882	/*
883	* Shadow - AMD64 mode
884	*/
885	#define PGM_SHW_TYPE PGM_TYPE_AMD64
886	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
887	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
888	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
889	#include "PGMShw.h"
890
891	#ifdef VBOX_WITH_64_BITS_GUESTS
892	/* Guest - AMD64 mode */
893	# define PGM_GST_TYPE PGM_TYPE_AMD64
894	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
895	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
896	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
897	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
898	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
899	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
900	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
903	# include "PGMBth.h"
904	# include "PGMGstDefs.h"
905	# include "PGMGst.h"
906	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	# undef BTH_PGMPOOLKIND_PT_FOR_PT
908	# undef BTH_PGMPOOLKIND_ROOT
909	# undef PGM_BTH_NAME
910	# undef PGM_BTH_NAME_RC_STR
911	# undef PGM_BTH_NAME_R0_STR
912	# undef PGM_GST_TYPE
913	# undef PGM_GST_NAME
914	# undef PGM_GST_NAME_RC_STR
915	# undef PGM_GST_NAME_R0_STR
916	#endif /* VBOX_WITH_64_BITS_GUESTS */
917
918	#undef PGM_SHW_TYPE
919	#undef PGM_SHW_NAME
920	#undef PGM_SHW_NAME_RC_STR
921	#undef PGM_SHW_NAME_R0_STR
922
923
924	/*
925	* Shadow - Nested paging mode
926	*/
927	#define PGM_SHW_TYPE PGM_TYPE_NESTED
928	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
929	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
930	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
931	#include "PGMShw.h"
932
933	/* Guest - real mode */
934	#define PGM_GST_TYPE PGM_TYPE_REAL
935	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
936	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
937	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
938	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
939	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
940	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
941	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
942	#include "PGMGstDefs.h"
943	#include "PGMBth.h"
944	#undef BTH_PGMPOOLKIND_PT_FOR_PT
945	#undef PGM_BTH_NAME
946	#undef PGM_BTH_NAME_RC_STR
947	#undef PGM_BTH_NAME_R0_STR
948	#undef PGM_GST_TYPE
949	#undef PGM_GST_NAME
950	#undef PGM_GST_NAME_RC_STR
951	#undef PGM_GST_NAME_R0_STR
952
953	/* Guest - protected mode */
954	#define PGM_GST_TYPE PGM_TYPE_PROT
955	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
956	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
957	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
958	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
959	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
960	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
961	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
962	#include "PGMGstDefs.h"
963	#include "PGMBth.h"
964	#undef BTH_PGMPOOLKIND_PT_FOR_PT
965	#undef PGM_BTH_NAME
966	#undef PGM_BTH_NAME_RC_STR
967	#undef PGM_BTH_NAME_R0_STR
968	#undef PGM_GST_TYPE
969	#undef PGM_GST_NAME
970	#undef PGM_GST_NAME_RC_STR
971	#undef PGM_GST_NAME_R0_STR
972
973	/* Guest - 32-bit mode */
974	#define PGM_GST_TYPE PGM_TYPE_32BIT
975	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
976	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
977	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
978	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
979	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
980	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
981	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
982	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
986	#undef BTH_PGMPOOLKIND_PT_FOR_PT
987	#undef PGM_BTH_NAME
988	#undef PGM_BTH_NAME_RC_STR
989	#undef PGM_BTH_NAME_R0_STR
990	#undef PGM_GST_TYPE
991	#undef PGM_GST_NAME
992	#undef PGM_GST_NAME_RC_STR
993	#undef PGM_GST_NAME_R0_STR
994
995	/* Guest - PAE mode */
996	#define PGM_GST_TYPE PGM_TYPE_PAE
997	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
998	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
999	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1000	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1001	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1002	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1003	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1004	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1005	#include "PGMGstDefs.h"
1006	#include "PGMBth.h"
1007	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1008	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1009	#undef PGM_BTH_NAME
1010	#undef PGM_BTH_NAME_RC_STR
1011	#undef PGM_BTH_NAME_R0_STR
1012	#undef PGM_GST_TYPE
1013	#undef PGM_GST_NAME
1014	#undef PGM_GST_NAME_RC_STR
1015	#undef PGM_GST_NAME_R0_STR
1016
1017	#ifdef VBOX_WITH_64_BITS_GUESTS
1018	/* Guest - AMD64 mode */
1019	# define PGM_GST_TYPE PGM_TYPE_AMD64
1020	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1021	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1022	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1023	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1024	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1025	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1026	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1027	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1028	# include "PGMGstDefs.h"
1029	# include "PGMBth.h"
1030	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1031	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1032	# undef PGM_BTH_NAME
1033	# undef PGM_BTH_NAME_RC_STR
1034	# undef PGM_BTH_NAME_R0_STR
1035	# undef PGM_GST_TYPE
1036	# undef PGM_GST_NAME
1037	# undef PGM_GST_NAME_RC_STR
1038	# undef PGM_GST_NAME_R0_STR
1039	#endif /* VBOX_WITH_64_BITS_GUESTS */
1040
1041	#undef PGM_SHW_TYPE
1042	#undef PGM_SHW_NAME
1043	#undef PGM_SHW_NAME_RC_STR
1044	#undef PGM_SHW_NAME_R0_STR
1045
1046
1047	/*
1048	* Shadow - EPT
1049	*/
1050	#define PGM_SHW_TYPE PGM_TYPE_EPT
1051	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1052	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1053	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1054	#include "PGMShw.h"
1055
1056	/* Guest - real mode */
1057	#define PGM_GST_TYPE PGM_TYPE_REAL
1058	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1059	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1060	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1061	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1062	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1063	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1064	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1065	#include "PGMGstDefs.h"
1066	#include "PGMBth.h"
1067	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1068	#undef PGM_BTH_NAME
1069	#undef PGM_BTH_NAME_RC_STR
1070	#undef PGM_BTH_NAME_R0_STR
1071	#undef PGM_GST_TYPE
1072	#undef PGM_GST_NAME
1073	#undef PGM_GST_NAME_RC_STR
1074	#undef PGM_GST_NAME_R0_STR
1075
1076	/* Guest - protected mode */
1077	#define PGM_GST_TYPE PGM_TYPE_PROT
1078	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1079	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1080	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1081	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1082	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1083	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1084	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1085	#include "PGMGstDefs.h"
1086	#include "PGMBth.h"
1087	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1088	#undef PGM_BTH_NAME
1089	#undef PGM_BTH_NAME_RC_STR
1090	#undef PGM_BTH_NAME_R0_STR
1091	#undef PGM_GST_TYPE
1092	#undef PGM_GST_NAME
1093	#undef PGM_GST_NAME_RC_STR
1094	#undef PGM_GST_NAME_R0_STR
1095
1096	/* Guest - 32-bit mode */
1097	#define PGM_GST_TYPE PGM_TYPE_32BIT
1098	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1099	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1100	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1101	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1102	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1103	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1104	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1105	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1109	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1110	#undef PGM_BTH_NAME
1111	#undef PGM_BTH_NAME_RC_STR
1112	#undef PGM_BTH_NAME_R0_STR
1113	#undef PGM_GST_TYPE
1114	#undef PGM_GST_NAME
1115	#undef PGM_GST_NAME_RC_STR
1116	#undef PGM_GST_NAME_R0_STR
1117
1118	/* Guest - PAE mode */
1119	#define PGM_GST_TYPE PGM_TYPE_PAE
1120	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1121	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1122	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1123	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1124	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1125	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1126	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1127	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1128	#include "PGMGstDefs.h"
1129	#include "PGMBth.h"
1130	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1131	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1132	#undef PGM_BTH_NAME
1133	#undef PGM_BTH_NAME_RC_STR
1134	#undef PGM_BTH_NAME_R0_STR
1135	#undef PGM_GST_TYPE
1136	#undef PGM_GST_NAME
1137	#undef PGM_GST_NAME_RC_STR
1138	#undef PGM_GST_NAME_R0_STR
1139
1140	#ifdef VBOX_WITH_64_BITS_GUESTS
1141	/* Guest - AMD64 mode */
1142	# define PGM_GST_TYPE PGM_TYPE_AMD64
1143	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1144	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1145	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1146	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1147	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1148	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1149	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1150	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1151	# include "PGMGstDefs.h"
1152	# include "PGMBth.h"
1153	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1154	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1155	# undef PGM_BTH_NAME
1156	# undef PGM_BTH_NAME_RC_STR
1157	# undef PGM_BTH_NAME_R0_STR
1158	# undef PGM_GST_TYPE
1159	# undef PGM_GST_NAME
1160	# undef PGM_GST_NAME_RC_STR
1161	# undef PGM_GST_NAME_R0_STR
1162	#endif /* VBOX_WITH_64_BITS_GUESTS */
1163
1164	#undef PGM_SHW_TYPE
1165	#undef PGM_SHW_NAME
1166	#undef PGM_SHW_NAME_RC_STR
1167	#undef PGM_SHW_NAME_R0_STR
1168
1169
1170
1171	/**
1172	* Initiates the paging of VM.
1173	*
1174	* @returns VBox status code.
1175	* @param pVM Pointer to VM structure.
1176	*/
1177	VMMR3DECL(int) PGMR3Init(PVM pVM)
1178	{
1179	LogFlow(("PGMR3Init:\n"));
1180	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1181	int rc;
1182
1183	/*
1184	* Assert alignment and sizes.
1185	*/
1186	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1187	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1188	AssertCompileMemberAlignment(PGM, CritSect, sizeof(uintptr_t));
1189
1190	/*
1191	* Init the structure.
1192	*/
1193	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1194	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1195
1196	/* Init the per-CPU part. */
1197	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1198	{
1199	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1200	PPGMCPU pPGM = &pVCpu->pgm.s;
1201
1202	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1203	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1204	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1205
1206	pPGM->enmShadowMode = PGMMODE_INVALID;
1207	pPGM->enmGuestMode = PGMMODE_INVALID;
1208
1209	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1210
1211	pPGM->pGstPaePdptR3 = NULL;
1212	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1213	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1214	#endif
1215	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1216	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1217	{
1218	pPGM->apGstPaePDsR3[i] = NULL;
1219	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1220	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1221	#endif
1222	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1223	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1224	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1225	}
1226
1227	pPGM->fA20Enabled = true;
1228	}
1229
1230	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1231	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1232	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1233
1234	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1235	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1236	true
1237	#else
1238	false
1239	#endif
1240	);
1241	AssertLogRelRCReturn(rc, rc);
1242
1243	#if HC_ARCH_BITS == 64 \|\| 1 /** @todo 4GB/32-bit: remove \|\| 1 later and adjust the limit. */
1244	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1245	#else
1246	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1247	#endif
1248	AssertLogRelRCReturn(rc, rc);
1249	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1250	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1251
1252	/*
1253	* Get the configured RAM size - to estimate saved state size.
1254	*/
1255	uint64_t cbRam;
1256	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1257	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1258	cbRam = 0;
1259	else if (RT_SUCCESS(rc))
1260	{
1261	if (cbRam < PAGE_SIZE)
1262	cbRam = 0;
1263	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1264	}
1265	else
1266	{
1267	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1268	return rc;
1269	}
1270
1271	/*
1272	* Register callbacks, string formatters and the saved state data unit.
1273	*/
1274	#ifdef VBOX_STRICT
1275	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1276	#endif
1277	PGMRegisterStringFormatTypes();
1278
1279	rc = pgmR3InitSavedState(pVM, cbRam);
1280	if (RT_FAILURE(rc))
1281	return rc;
1282
1283	/*
1284	* Initialize the PGM critical section and flush the phys TLBs
1285	*/
1286	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, RT_SRC_POS, "PGM");
1287	AssertRCReturn(rc, rc);
1288
1289	PGMR3PhysChunkInvalidateTLB(pVM);
1290	PGMPhysInvalidatePageMapTLB(pVM);
1291
1292	/*
1293	* For the time being we sport a full set of handy pages in addition to the base
1294	* memory to simplify things.
1295	*/
1296	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1297	AssertRCReturn(rc, rc);
1298
1299	/*
1300	* Trees
1301	*/
1302	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1303	if (RT_SUCCESS(rc))
1304	{
1305	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1306	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1307
1308	/*
1309	* Alocate the zero page.
1310	*/
1311	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1312	}
1313	if (RT_SUCCESS(rc))
1314	{
1315	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1316	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1317	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1318	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1319
1320	/*
1321	* Init the paging.
1322	*/
1323	rc = pgmR3InitPaging(pVM);
1324	}
1325	if (RT_SUCCESS(rc))
1326	{
1327	/*
1328	* Init the page pool.
1329	*/
1330	rc = pgmR3PoolInit(pVM);
1331	}
1332	if (RT_SUCCESS(rc))
1333	{
1334	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1335	{
1336	PVMCPU pVCpu = &pVM->aCpus[i];
1337	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1338	if (RT_FAILURE(rc))
1339	break;
1340	}
1341	}
1342
1343	if (RT_SUCCESS(rc))
1344	{
1345	/*
1346	* Info & statistics
1347	*/
1348	DBGFR3InfoRegisterInternal(pVM, "mode",
1349	"Shows the current paging mode. "
1350	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing's given.",
1351	pgmR3InfoMode);
1352	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1353	"Dumps all the entries in the top level paging table. No arguments.",
1354	pgmR3InfoCr3);
1355	DBGFR3InfoRegisterInternal(pVM, "phys",
1356	"Dumps all the physical address ranges. No arguments.",
1357	pgmR3PhysInfo);
1358	DBGFR3InfoRegisterInternal(pVM, "handlers",
1359	"Dumps physical, virtual and hyper virtual handlers. "
1360	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1361	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1362	pgmR3InfoHandlers);
1363	DBGFR3InfoRegisterInternal(pVM, "mappings",
1364	"Dumps guest mappings.",
1365	pgmR3MapInfo);
1366
1367	pgmR3InitStats(pVM);
1368
1369	#ifdef VBOX_WITH_DEBUGGER
1370	/*
1371	* Debugger commands.
1372	*/
1373	static bool s_fRegisteredCmds = false;
1374	if (!s_fRegisteredCmds)
1375	{
1376	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1377	if (RT_SUCCESS(rc2))
1378	s_fRegisteredCmds = true;
1379	}
1380	#endif
1381	return VINF_SUCCESS;
1382	}
1383
1384	/* Almost no cleanup necessary, MM frees all memory. */
1385	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1386
1387	return rc;
1388	}
1389
1390
1391	/**
1392	* Initializes the per-VCPU PGM.
1393	*
1394	* @returns VBox status code.
1395	* @param pVM The VM to operate on.
1396	*/
1397	VMMR3DECL(int) PGMR3InitCPU(PVM pVM)
1398	{
1399	LogFlow(("PGMR3InitCPU\n"));
1400	return VINF_SUCCESS;
1401	}
1402
1403
1404	/**
1405	* Init paging.
1406	*
1407	* Since we need to check what mode the host is operating in before we can choose
1408	* the right paging functions for the host we have to delay this until R0 has
1409	* been initialized.
1410	*
1411	* @returns VBox status code.
1412	* @param pVM VM handle.
1413	*/
1414	static int pgmR3InitPaging(PVM pVM)
1415	{
1416	/*
1417	* Force a recalculation of modes and switcher so everyone gets notified.
1418	*/
1419	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1420	{
1421	PVMCPU pVCpu = &pVM->aCpus[i];
1422
1423	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1424	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1425	}
1426
1427	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1428
1429	/*
1430	* Allocate static mapping space for whatever the cr3 register
1431	* points to and in the case of PAE mode to the 4 PDs.
1432	*/
1433	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1434	if (RT_FAILURE(rc))
1435	{
1436	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1437	return rc;
1438	}
1439	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1440
1441	/*
1442	* Allocate pages for the three possible intermediate contexts
1443	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1444	* for the sake of simplicity. The AMD64 uses the PAE for the
1445	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1446	*
1447	* We assume that two page tables will be enought for the core code
1448	* mappings (HC virtual and identity).
1449	*/
1450	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1451	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1452	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1453	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1454	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1455	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1456	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1457	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1458	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1459	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1460	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1461	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1462
1463	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1464	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1465	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1466	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1467	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1468	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1469
1470	/*
1471	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1472	*/
1473	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1474	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1475	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1476
1477	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1478	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1479
1480	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1481	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1482	{
1483	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1484	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1485	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1486	}
1487
1488	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1489	{
1490	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1491	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1492	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1493	}
1494
1495	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1496	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1497	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1498	\| HCPhysInterPaePDPT64;
1499
1500	/*
1501	* Initialize paging workers and mode from current host mode
1502	* and the guest running in real mode.
1503	*/
1504	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1505	switch (pVM->pgm.s.enmHostMode)
1506	{
1507	case SUPPAGINGMODE_32_BIT:
1508	case SUPPAGINGMODE_32_BIT_GLOBAL:
1509	case SUPPAGINGMODE_PAE:
1510	case SUPPAGINGMODE_PAE_GLOBAL:
1511	case SUPPAGINGMODE_PAE_NX:
1512	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1513	break;
1514
1515	case SUPPAGINGMODE_AMD64:
1516	case SUPPAGINGMODE_AMD64_GLOBAL:
1517	case SUPPAGINGMODE_AMD64_NX:
1518	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1519	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1520	if (ARCH_BITS != 64)
1521	{
1522	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1523	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1524	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1525	}
1526	#endif
1527	break;
1528	default:
1529	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1530	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1531	}
1532	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1533	if (RT_SUCCESS(rc))
1534	{
1535	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1536	#if HC_ARCH_BITS == 64
1537	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1538	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1539	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1540	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1541	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1542	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1543	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1544	#endif
1545
1546	return VINF_SUCCESS;
1547	}
1548
1549	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1550	return rc;
1551	}
1552
1553
1554	/**
1555	* Init statistics
1556	*/
1557	static void pgmR3InitStats(PVM pVM)
1558	{
1559	PPGM pPGM = &pVM->pgm.s;
1560	int rc;
1561
1562	/* Common - misc variables */
1563	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1564	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1565	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1566	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1567	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1568	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1569	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1570	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1571	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1572	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1573	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1574	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1575	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1576
1577	/* Live save */
1578	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1579	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1580	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1581	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1582	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1583	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1584	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1585	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1586	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1587	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1588	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1589	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1590	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1591	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1592	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1593	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1594	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1595	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1596
1597	#ifdef VBOX_WITH_STATISTICS
1598
1599	# define PGM_REG_COUNTER(a, b, c) \
1600	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1601	AssertRC(rc);
1602
1603	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1604	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1605	AssertRC(rc);
1606
1607	# define PGM_REG_PROFILE(a, b, c) \
1608	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1609	AssertRC(rc);
1610
1611	PGM_REG_COUNTER(&pPGM->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1612	PGM_REG_PROFILE(&pPGM->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1613	PGM_REG_COUNTER(&pPGM->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1614	PGM_REG_COUNTER_BYTES(&pPGM->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1615	PGM_REG_COUNTER(&pPGM->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1616	PGM_REG_COUNTER_BYTES(&pPGM->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1617	PGM_REG_COUNTER(&pPGM->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1618	PGM_REG_COUNTER_BYTES(&pPGM->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1619	PGM_REG_COUNTER(&pPGM->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1620	PGM_REG_COUNTER_BYTES(&pPGM->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1621
1622	PGM_REG_COUNTER(&pPGM->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1623	PGM_REG_COUNTER(&pPGM->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1624	PGM_REG_COUNTER(&pPGM->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1625	PGM_REG_COUNTER(&pPGM->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1626	PGM_REG_COUNTER(&pPGM->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1627	PGM_REG_COUNTER(&pPGM->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1628	PGM_REG_COUNTER(&pPGM->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1629	PGM_REG_COUNTER(&pPGM->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1630	PGM_REG_COUNTER(&pPGM->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1631	PGM_REG_COUNTER(&pPGM->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1632
1633	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1634	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1635	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1636	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1637
1638	PGM_REG_COUNTER(&pPGM->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1639	PGM_REG_COUNTER(&pPGM->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1640	PGM_REG_PROFILE(&pPGM->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1641	PGM_REG_PROFILE(&pPGM->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1642
1643	PGM_REG_COUNTER(&pPGM->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1644	PGM_REG_COUNTER(&pPGM->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1645	/// @todo PGM_REG_COUNTER(&pPGM->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1646	PGM_REG_COUNTER(&pPGM->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1647	PGM_REG_COUNTER(&pPGM->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1648	/// @todo PGM_REG_COUNTER(&pPGM->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1649
1650	PGM_REG_COUNTER(&pPGM->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1651	PGM_REG_COUNTER_BYTES(&pPGM->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1652	PGM_REG_COUNTER(&pPGM->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1653	PGM_REG_COUNTER_BYTES(&pPGM->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1654	PGM_REG_COUNTER(&pPGM->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1655	PGM_REG_COUNTER_BYTES(&pPGM->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1656	PGM_REG_COUNTER(&pPGM->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1657	PGM_REG_COUNTER_BYTES(&pPGM->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1658
1659	/* GC only: */
1660	PGM_REG_COUNTER(&pPGM->StatRCDynMapCacheHits, "/PGM/RC/DynMapCache/Hits" , "Number of dynamic page mapping cache hits.");
1661	PGM_REG_COUNTER(&pPGM->StatRCDynMapCacheMisses, "/PGM/RC/DynMapCache/Misses" , "Number of dynamic page mapping cache misses.");
1662	PGM_REG_COUNTER(&pPGM->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1663	PGM_REG_COUNTER(&pPGM->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1664
1665	PGM_REG_COUNTER(&pPGM->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1666	PGM_REG_COUNTER_BYTES(&pPGM->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1667	PGM_REG_COUNTER(&pPGM->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1668	PGM_REG_COUNTER_BYTES(&pPGM->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1669	PGM_REG_COUNTER(&pPGM->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1670	PGM_REG_COUNTER_BYTES(&pPGM->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1671	PGM_REG_COUNTER(&pPGM->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1672	PGM_REG_COUNTER_BYTES(&pPGM->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1673
1674	# ifdef PGMPOOL_WITH_GCPHYS_TRACKING
1675	PGM_REG_COUNTER(&pPGM->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1676	PGM_REG_COUNTER(&pPGM->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1677	PGM_REG_COUNTER(&pPGM->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1678	PGM_REG_COUNTER(&pPGM->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1679	PGM_REG_COUNTER(&pPGM->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1680	PGM_REG_PROFILE(&pPGM->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1681	# endif
1682
1683	# undef PGM_REG_COUNTER
1684	# undef PGM_REG_PROFILE
1685	#endif
1686
1687	/*
1688	* Note! The layout below matches the member layout exactly!
1689	*/
1690
1691	/*
1692	* Common - stats
1693	*/
1694	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1695	{
1696	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1697
1698	#define PGM_REG_COUNTER(a, b, c) \
1699	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1700	AssertRC(rc);
1701	#define PGM_REG_PROFILE(a, b, c) \
1702	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1703	AssertRC(rc);
1704
1705	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1706
1707	#ifdef VBOX_WITH_STATISTICS
1708
1709	# if 0 /* rarely useful; leave for debugging. */
1710	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1711	STAMR3RegisterF(pVM, &pPgmCpu->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1712	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1713	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPagePD); j++)
1714	STAMR3RegisterF(pVM, &pPgmCpu->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1715	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1716	# endif
1717	/* R0 only: */
1718	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapMigrateInvlPg, "/PGM/CPU%u/R0/DynMapMigrateInvlPg", "invlpg count in PGMDynMapMigrateAutoSet.");
1719	PGM_REG_PROFILE(&pPgmCpu->StatR0DynMapGCPageInl, "/PGM/CPU%u/R0/DynMapPageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1720	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapGCPageInlHits, "/PGM/CPU%u/R0/DynMapPageGCPageInl/Hits", "Hash table lookup hits.");
1721	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapGCPageInlMisses, "/PGM/CPU%u/R0/DynMapPageGCPageInl/Misses", "Misses that falls back to code common with PGMDynMapHCPage.");
1722	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapGCPageInlRamHits, "/PGM/CPU%u/R0/DynMapPageGCPageInl/RamHits", "1st ram range hits.");
1723	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapGCPageInlRamMisses, "/PGM/CPU%u/R0/DynMapPageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1724	PGM_REG_PROFILE(&pPgmCpu->StatR0DynMapHCPageInl, "/PGM/CPU%u/R0/DynMapPageHCPageInl", "Calls to pgmR0DynMapHCPageInlined.");
1725	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapHCPageInlHits, "/PGM/CPU%u/R0/DynMapPageHCPageInl/Hits", "Hash table lookup hits.");
1726	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapHCPageInlMisses, "/PGM/CPU%u/R0/DynMapPageHCPageInl/Misses", "Misses that falls back to code common with PGMDynMapHCPage.");
1727	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPage, "/PGM/CPU%u/R0/DynMapPage", "Calls to pgmR0DynMapPage");
1728	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapSetOptimize, "/PGM/CPU%u/R0/DynMapPage/SetOptimize", "Calls to pgmDynMapOptimizeAutoSet.");
1729	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapSetSearchFlushes, "/PGM/CPU%u/R0/DynMapPage/SetSearchFlushes","Set search restorting to subset flushes.");
1730	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapSetSearchHits, "/PGM/CPU%u/R0/DynMapPage/SetSearchHits", "Set search hits.");
1731	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapSetSearchMisses, "/PGM/CPU%u/R0/DynMapPage/SetSearchMisses", "Set search misses.");
1732	PGM_REG_PROFILE(&pPgmCpu->StatR0DynMapHCPage, "/PGM/CPU%u/R0/DynMapPage/HCPage", "Calls to PGMDynMapHCPage (ring-0).");
1733	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageHits0, "/PGM/CPU%u/R0/DynMapPage/Hits0", "Hits at iPage+0");
1734	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageHits1, "/PGM/CPU%u/R0/DynMapPage/Hits1", "Hits at iPage+1");
1735	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageHits2, "/PGM/CPU%u/R0/DynMapPage/Hits2", "Hits at iPage+2");
1736	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageInvlPg, "/PGM/CPU%u/R0/DynMapPage/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1737	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageSlow, "/PGM/CPU%u/R0/DynMapPage/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1738	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageSlowLoopHits, "/PGM/CPU%u/R0/DynMapPage/SlowLoopHits" , "Hits in the loop path.");
1739	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageSlowLoopMisses, "/PGM/CPU%u/R0/DynMapPage/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1740	//PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMapPage/SlowLostHits", "Lost hits.");
1741	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapSubsets, "/PGM/CPU%u/R0/Subsets", "Times PGMDynMapPushAutoSubset was called.");
1742	PGM_REG_COUNTER(&pPgmCpu->StatR0DynMapPopFlushes, "/PGM/CPU%u/R0/SubsetPopFlushes", "Times PGMDynMapPopAutoSubset flushes the subset.");
1743	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[0], "/PGM/CPU%u/R0/SetSize000..09", "00-09% filled");
1744	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[1], "/PGM/CPU%u/R0/SetSize010..19", "10-19% filled");
1745	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[2], "/PGM/CPU%u/R0/SetSize020..29", "20-29% filled");
1746	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[3], "/PGM/CPU%u/R0/SetSize030..39", "30-39% filled");
1747	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[4], "/PGM/CPU%u/R0/SetSize040..49", "40-49% filled");
1748	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[5], "/PGM/CPU%u/R0/SetSize050..59", "50-59% filled");
1749	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[6], "/PGM/CPU%u/R0/SetSize060..69", "60-69% filled");
1750	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[7], "/PGM/CPU%u/R0/SetSize070..79", "70-79% filled");
1751	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[8], "/PGM/CPU%u/R0/SetSize080..89", "80-89% filled");
1752	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[9], "/PGM/CPU%u/R0/SetSize090..99", "90-99% filled");
1753	PGM_REG_COUNTER(&pPgmCpu->aStatR0DynMapSetSize[10], "/PGM/CPU%u/R0/SetSize100", "100% filled");
1754
1755	/* RZ only: */
1756	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1757	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTimeCheckPageFault, "/PGM/CPU%u/RZ/Trap0e/Time/CheckPageFault", "Profiling of checking for dirty/access emulation faults.");
1758	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTimeSyncPT, "/PGM/CPU%u/RZ/Trap0e/Time/SyncPT", "Profiling of lazy page table syncing.");
1759	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTimeMapping, "/PGM/CPU%u/RZ/Trap0e/Time/Mapping", "Profiling of checking virtual mappings.");
1760	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTimeOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time/OutOfSync", "Profiling of out of sync page handling.");
1761	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTimeHandlers, "/PGM/CPU%u/RZ/Trap0e/Time/Handlers", "Profiling of checking handlers.");
1762	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1763	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1764	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1765	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1766	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1767	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1768	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1769	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1770	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1771	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1772	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1773	PGM_REG_PROFILE(&pPgmCpu->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1774	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1775	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1776	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1777	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersPhysical, "/PGM/CPU%u/RZ/Trap0e/Handlers/Physical", "Number of traps due to physical access handlers.");
1778	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1779	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1780	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1781	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1782	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1783	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1784	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1785	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1786	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1787	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1788	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1789	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1790	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1791	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1792	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1793	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1794	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1795	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eGuestPFUnh, "/PGM/CPU%u/RZ/Trap0e/GuestPF/Unhandled", "Number of real guest page faults from the 'unhandled' case.");
1796	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1797	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1798	PGM_REG_COUNTER(&pPgmCpu->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1799	#if 0 /* rarely useful; leave for debugging. */
1800	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatRZTrap0ePD); j++)
1801	STAMR3RegisterF(pVM, &pPgmCpu->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1802	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1803	#endif
1804	PGM_REG_COUNTER(&pPgmCpu->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1805	PGM_REG_COUNTER(&pPgmCpu->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1806	PGM_REG_COUNTER(&pPgmCpu->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1807	PGM_REG_COUNTER(&pPgmCpu->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1808	PGM_REG_COUNTER(&pPgmCpu->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1809
1810	/* HC only: */
1811
1812	/* RZ & R3: */
1813	PGM_REG_PROFILE(&pPgmCpu->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1814	PGM_REG_PROFILE(&pPgmCpu->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1815	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1816	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1817	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1818	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1819	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1820	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1821	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1822	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1823	PGM_REG_PROFILE(&pPgmCpu->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1824	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1825	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1826	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1827	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1828	PGM_REG_COUNTER(&pPgmCpu->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1829	PGM_REG_COUNTER(&pPgmCpu->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1830	PGM_REG_PROFILE(&pPgmCpu->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1831	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1832	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1833	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1834	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1835	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1836	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1837	PGM_REG_COUNTER(&pPgmCpu->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1838	PGM_REG_COUNTER(&pPgmCpu->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1839	PGM_REG_PROFILE(&pPgmCpu->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1840	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1841	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1842	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1843	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1844	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1845	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1846	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1847	PGM_REG_COUNTER(&pPgmCpu->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1848	PGM_REG_COUNTER(&pPgmCpu->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1849	PGM_REG_COUNTER(&pPgmCpu->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1850	PGM_REG_COUNTER(&pPgmCpu->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1851	PGM_REG_COUNTER(&pPgmCpu->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1852	PGM_REG_PROFILE(&pPgmCpu->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
1853	PGM_REG_PROFILE(&pPgmCpu->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1854	PGM_REG_COUNTER(&pPgmCpu->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1855	PGM_REG_COUNTER(&pPgmCpu->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1856	PGM_REG_COUNTER(&pPgmCpu->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1857	PGM_REG_COUNTER(&pPgmCpu->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1858	PGM_REG_PROFILE(&pPgmCpu->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1859
1860	PGM_REG_PROFILE(&pPgmCpu->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1861	PGM_REG_PROFILE(&pPgmCpu->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1862	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1863	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1864	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1865	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1866	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1867	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1868	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1869	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1870	PGM_REG_PROFILE(&pPgmCpu->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1871	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1872	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
1873	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
1874	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1875	PGM_REG_COUNTER(&pPgmCpu->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1876	PGM_REG_COUNTER(&pPgmCpu->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1877	PGM_REG_PROFILE(&pPgmCpu->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1878	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1879	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1880	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1881	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1882	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1883	PGM_REG_COUNTER(&pPgmCpu->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1884	PGM_REG_COUNTER(&pPgmCpu->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1885	PGM_REG_PROFILE(&pPgmCpu->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
1886	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1887	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1888	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1889	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1890	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1891	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1892	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1893	PGM_REG_COUNTER(&pPgmCpu->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1894	PGM_REG_COUNTER(&pPgmCpu->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1895	PGM_REG_COUNTER(&pPgmCpu->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1896	PGM_REG_PROFILE(&pPgmCpu->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
1897	PGM_REG_PROFILE(&pPgmCpu->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1898	PGM_REG_COUNTER(&pPgmCpu->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1899	PGM_REG_COUNTER(&pPgmCpu->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1900	PGM_REG_COUNTER(&pPgmCpu->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1901	PGM_REG_COUNTER(&pPgmCpu->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1902	PGM_REG_PROFILE(&pPgmCpu->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1903	#endif /* VBOX_WITH_STATISTICS */
1904
1905	#undef PGM_REG_PROFILE
1906	#undef PGM_REG_COUNTER
1907
1908	}
1909	}
1910
1911
1912	/**
1913	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1914	*
1915	* The dynamic mapping area will also be allocated and initialized at this
1916	* time. We could allocate it during PGMR3Init of course, but the mapping
1917	* wouldn't be allocated at that time preventing us from setting up the
1918	* page table entries with the dummy page.
1919	*
1920	* @returns VBox status code.
1921	* @param pVM VM handle.
1922	*/
1923	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1924	{
1925	RTGCPTR GCPtr;
1926	int rc;
1927
1928	/*
1929	* Reserve space for the dynamic mappings.
1930	*/
1931	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
1932	if (RT_SUCCESS(rc))
1933	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
1934
1935	if ( RT_SUCCESS(rc)
1936	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
1937	{
1938	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
1939	if (RT_SUCCESS(rc))
1940	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
1941	}
1942	if (RT_SUCCESS(rc))
1943	{
1944	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
1945	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1946	}
1947	return rc;
1948	}
1949
1950
1951	/**
1952	* Ring-3 init finalizing.
1953	*
1954	* @returns VBox status code.
1955	* @param pVM The VM handle.
1956	*/
1957	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
1958	{
1959	int rc;
1960
1961	/*
1962	* Reserve space for the dynamic mappings.
1963	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
1964	*/
1965	/* get the pointer to the page table entries. */
1966	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
1967	AssertRelease(pMapping);
1968	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
1969	const unsigned iPT = off >> X86_PD_SHIFT;
1970	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
1971	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
1972	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
1973
1974	/* init cache */
1975	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
1976	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aHCPhysDynPageMapCache); i++)
1977	pVM->pgm.s.aHCPhysDynPageMapCache[i] = HCPhysDummy;
1978
1979	for (unsigned i = 0; i < MM_HYPER_DYNAMIC_SIZE; i += PAGE_SIZE)
1980	{
1981	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + i, HCPhysDummy, PAGE_SIZE, 0);
1982	AssertRCReturn(rc, rc);
1983	}
1984
1985	/*
1986	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
1987	* Intel only goes up to 36 bits, so we stick to 36 as well.
1988	*/
1989	/** @todo How to test for the 40 bits support? Long mode seems to be the test criterium. */
1990	uint32_t u32Dummy, u32Features;
1991	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
1992
1993	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
1994	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(36) - 1;
1995	else
1996	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
1997
1998	/*
1999	* Allocate memory if we're supposed to do that.
2000	*/
2001	if (pVM->pgm.s.fRamPreAlloc)
2002	rc = pgmR3PhysRamPreAllocate(pVM);
2003
2004	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2005	return rc;
2006	}
2007
2008
2009	/**
2010	* Applies relocations to data and code managed by this component.
2011	*
2012	* This function will be called at init and whenever the VMM need to relocate it
2013	* self inside the GC.
2014	*
2015	* @param pVM The VM.
2016	* @param offDelta Relocation delta relative to old location.
2017	*/
2018	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2019	{
2020	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2021
2022	/*
2023	* Paging stuff.
2024	*/
2025	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2026
2027	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2028
2029	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2030	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2031	{
2032	PVMCPU pVCpu = &pVM->aCpus[i];
2033
2034	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2035
2036	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2037	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2038	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2039	}
2040
2041	/*
2042	* Trees.
2043	*/
2044	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2045
2046	/*
2047	* Ram ranges.
2048	*/
2049	if (pVM->pgm.s.pRamRangesR3)
2050	{
2051	/* Update the pSelfRC pointers and relink them. */
2052	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2053	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2054	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2055	pgmR3PhysRelinkRamRanges(pVM);
2056	}
2057
2058	/*
2059	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2060	* be mapped and thus not included in the above exercise.
2061	*/
2062	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2063	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2064	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2065
2066	/*
2067	* Update the two page directories with all page table mappings.
2068	* (One or more of them have changed, that's why we're here.)
2069	*/
2070	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2071	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2072	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2073
2074	/* Relocate GC addresses of Page Tables. */
2075	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2076	{
2077	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2078	{
2079	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2080	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2081	}
2082	}
2083
2084	/*
2085	* Dynamic page mapping area.
2086	*/
2087	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2088	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2089	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2090
2091	/*
2092	* The Zero page.
2093	*/
2094	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2095	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2096	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2097	#else
2098	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2099	#endif
2100
2101	/*
2102	* Physical and virtual handlers.
2103	*/
2104	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2105	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2106	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2107
2108	/*
2109	* The page pool.
2110	*/
2111	pgmR3PoolRelocate(pVM);
2112	}
2113
2114
2115	/**
2116	* Callback function for relocating a physical access handler.
2117	*
2118	* @returns 0 (continue enum)
2119	* @param pNode Pointer to a PGMPHYSHANDLER node.
2120	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2121	* not certain the delta will fit in a void pointer for all possible configs.
2122	*/
2123	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2124	{
2125	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2126	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2127	if (pHandler->pfnHandlerRC)
2128	pHandler->pfnHandlerRC += offDelta;
2129	if (pHandler->pvUserRC >= 0x10000)
2130	pHandler->pvUserRC += offDelta;
2131	return 0;
2132	}
2133
2134
2135	/**
2136	* Callback function for relocating a virtual access handler.
2137	*
2138	* @returns 0 (continue enum)
2139	* @param pNode Pointer to a PGMVIRTHANDLER node.
2140	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2141	* not certain the delta will fit in a void pointer for all possible configs.
2142	*/
2143	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2144	{
2145	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2146	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2147	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2148	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2149	Assert(pHandler->pfnHandlerRC);
2150	pHandler->pfnHandlerRC += offDelta;
2151	return 0;
2152	}
2153
2154
2155	/**
2156	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2157	*
2158	* @returns 0 (continue enum)
2159	* @param pNode Pointer to a PGMVIRTHANDLER node.
2160	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2161	* not certain the delta will fit in a void pointer for all possible configs.
2162	*/
2163	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2164	{
2165	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2166	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2167	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2168	Assert(pHandler->pfnHandlerRC);
2169	pHandler->pfnHandlerRC += offDelta;
2170	return 0;
2171	}
2172
2173
2174	/**
2175	* Resets a virtual CPU when unplugged.
2176	*
2177	* @param pVM The VM handle.
2178	* @param pVCpu The virtual CPU handle.
2179	*/
2180	VMMR3DECL(void) PGMR3ResetUnpluggedCpu(PVM pVM, PVMCPU pVCpu)
2181	{
2182	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2183	AssertRC(rc);
2184
2185	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2186	AssertRC(rc);
2187
2188	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2189
2190	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2191
2192	/*
2193	* Re-init other members.
2194	*/
2195	pVCpu->pgm.s.fA20Enabled = true;
2196
2197	/*
2198	* Clear the FFs PGM owns.
2199	*/
2200	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2201	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2202	}
2203
2204
2205	/**
2206	* The VM is being reset.
2207	*
2208	* For the PGM component this means that any PD write monitors
2209	* needs to be removed.
2210	*
2211	* @param pVM VM handle.
2212	*/
2213	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2214	{
2215	int rc;
2216
2217	LogFlow(("PGMR3Reset:\n"));
2218	VM_ASSERT_EMT(pVM);
2219
2220	pgmLock(pVM);
2221
2222	/*
2223	* Unfix any fixed mappings and disable CR3 monitoring.
2224	*/
2225	pVM->pgm.s.fMappingsFixed = false;
2226	pVM->pgm.s.fMappingsFixedRestored = false;
2227	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2228	pVM->pgm.s.cbMappingFixed = 0;
2229
2230	/*
2231	* Exit the guest paging mode before the pgm pool gets reset.
2232	* Important to clean up the amd64 case.
2233	*/
2234	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2235	{
2236	PVMCPU pVCpu = &pVM->aCpus[i];
2237	rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2238	AssertRC(rc);
2239	}
2240
2241	#ifdef DEBUG
2242	DBGFR3InfoLog(pVM, "mappings", NULL);
2243	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2244	#endif
2245
2246	/*
2247	* Switch mode back to real mode. (before resetting the pgm pool!)
2248	*/
2249	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2250	{
2251	PVMCPU pVCpu = &pVM->aCpus[i];
2252
2253	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2254	AssertRC(rc);
2255
2256	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2257	}
2258
2259	/*
2260	* Reset the shadow page pool.
2261	*/
2262	pgmR3PoolReset(pVM);
2263
2264	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2265	{
2266	PVMCPU pVCpu = &pVM->aCpus[i];
2267
2268	/*
2269	* Re-init other members.
2270	*/
2271	pVCpu->pgm.s.fA20Enabled = true;
2272
2273	/*
2274	* Clear the FFs PGM owns.
2275	*/
2276	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2277	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2278	}
2279
2280	/*
2281	* Reset (zero) RAM pages.
2282	*/
2283	rc = pgmR3PhysRamReset(pVM);
2284	if (RT_SUCCESS(rc))
2285	{
2286	/*
2287	* Reset (zero) shadow ROM pages.
2288	*/
2289	rc = pgmR3PhysRomReset(pVM);
2290	}
2291
2292	pgmUnlock(pVM);
2293	//return rc;
2294	AssertReleaseRC(rc);
2295	}
2296
2297
2298	#ifdef VBOX_STRICT
2299	/**
2300	* VM state change callback for clearing fNoMorePhysWrites after
2301	* a snapshot has been created.
2302	*/
2303	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2304	{
2305	if ( enmState == VMSTATE_RUNNING
2306	\|\| enmState == VMSTATE_RESUMING)
2307	pVM->pgm.s.fNoMorePhysWrites = false;
2308	}
2309	#endif
2310
2311
2312	/**
2313	* Terminates the PGM.
2314	*
2315	* @returns VBox status code.
2316	* @param pVM Pointer to VM structure.
2317	*/
2318	VMMR3DECL(int) PGMR3Term(PVM pVM)
2319	{
2320	PGMDeregisterStringFormatTypes();
2321	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
2322	}
2323
2324
2325	/**
2326	* Terminates the per-VCPU PGM.
2327	*
2328	* Termination means cleaning up and freeing all resources,
2329	* the VM it self is at this point powered off or suspended.
2330	*
2331	* @returns VBox status code.
2332	* @param pVM The VM to operate on.
2333	*/
2334	VMMR3DECL(int) PGMR3TermCPU(PVM pVM)
2335	{
2336	return 0;
2337	}
2338
2339
2340	/**
2341	* Show paging mode.
2342	*
2343	* @param pVM VM Handle.
2344	* @param pHlp The info helpers.
2345	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2346	*/
2347	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2348	{
2349	/* digest argument. */
2350	bool fGuest, fShadow, fHost;
2351	if (pszArgs)
2352	pszArgs = RTStrStripL(pszArgs);
2353	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2354	fShadow = fHost = fGuest = true;
2355	else
2356	{
2357	fShadow = fHost = fGuest = false;
2358	if (strstr(pszArgs, "guest"))
2359	fGuest = true;
2360	if (strstr(pszArgs, "shadow"))
2361	fShadow = true;
2362	if (strstr(pszArgs, "host"))
2363	fHost = true;
2364	}
2365
2366	/** @todo SMP support! */
2367	/* print info. */
2368	if (fGuest)
2369	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2370	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2371	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
2372	if (fShadow)
2373	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2374	if (fHost)
2375	{
2376	const char *psz;
2377	switch (pVM->pgm.s.enmHostMode)
2378	{
2379	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2380	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2381	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2382	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2383	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2384	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2385	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2386	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2387	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2388	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2389	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2390	default: psz = "unknown"; break;
2391	}
2392	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2393	}
2394	}
2395
2396
2397	/**
2398	* Dump registered MMIO ranges to the log.
2399	*
2400	* @param pVM VM Handle.
2401	* @param pHlp The info helpers.
2402	* @param pszArgs Arguments, ignored.
2403	*/
2404	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2405	{
2406	NOREF(pszArgs);
2407	pHlp->pfnPrintf(pHlp,
2408	"RAM ranges (pVM=%p)\n"
2409	"%.s %.s\n",
2410	pVM,
2411	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2412	sizeof(RTHCPTR) * 2, "pvHC ");
2413
2414	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2415	pHlp->pfnPrintf(pHlp,
2416	"%RGp-%RGp %RHv %s\n",
2417	pCur->GCPhys,
2418	pCur->GCPhysLast,
2419	pCur->pvR3,
2420	pCur->pszDesc);
2421	}
2422
2423	/**
2424	* Dump the page directory to the log.
2425	*
2426	* @param pVM VM Handle.
2427	* @param pHlp The info helpers.
2428	* @param pszArgs Arguments, ignored.
2429	*/
2430	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2431	{
2432	/** @todo SMP support!! */
2433	PVMCPU pVCpu = &pVM->aCpus[0];
2434
2435	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2436	/* Big pages supported? */
2437	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2438
2439	/* Global pages supported? */
2440	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2441
2442	NOREF(pszArgs);
2443
2444	/*
2445	* Get page directory addresses.
2446	*/
2447	PX86PD pPDSrc = pgmGstGet32bitPDPtr(&pVCpu->pgm.s);
2448	Assert(pPDSrc);
2449	Assert(PGMPhysGCPhys2R3PtrAssert(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2450
2451	/*
2452	* Iterate the page directory.
2453	*/
2454	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2455	{
2456	X86PDE PdeSrc = pPDSrc->a[iPD];
2457	if (PdeSrc.n.u1Present)
2458	{
2459	if (PdeSrc.b.u1Size && fPSE)
2460	pHlp->pfnPrintf(pHlp,
2461	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2462	iPD,
2463	pgmGstGet4MBPhysPage(&pVM->pgm.s, PdeSrc),
2464	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2465	else
2466	pHlp->pfnPrintf(pHlp,
2467	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2468	iPD,
2469	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2470	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2471	}
2472	}
2473	}
2474
2475
2476	/**
2477	* Service a VMMCALLRING3_PGM_LOCK call.
2478	*
2479	* @returns VBox status code.
2480	* @param pVM The VM handle.
2481	*/
2482	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2483	{
2484	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
2485	AssertRC(rc);
2486	return rc;
2487	}
2488
2489
2490	/**
2491	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2492	*
2493	* @returns PGM_TYPE_*.
2494	* @param pgmMode The mode value to convert.
2495	*/
2496	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2497	{
2498	switch (pgmMode)
2499	{
2500	case PGMMODE_REAL: return PGM_TYPE_REAL;
2501	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2502	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2503	case PGMMODE_PAE:
2504	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2505	case PGMMODE_AMD64:
2506	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2507	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2508	case PGMMODE_EPT: return PGM_TYPE_EPT;
2509	default:
2510	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2511	}
2512	}
2513
2514
2515	/**
2516	* Gets the index into the paging mode data array of a SHW+GST mode.
2517	*
2518	* @returns PGM::paPagingData index.
2519	* @param uShwType The shadow paging mode type.
2520	* @param uGstType The guest paging mode type.
2521	*/
2522	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2523	{
2524	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2525	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2526	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2527	+ (uGstType - PGM_TYPE_REAL);
2528	}
2529
2530
2531	/**
2532	* Gets the index into the paging mode data array of a SHW+GST mode.
2533	*
2534	* @returns PGM::paPagingData index.
2535	* @param enmShw The shadow paging mode.
2536	* @param enmGst The guest paging mode.
2537	*/
2538	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2539	{
2540	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2541	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2542	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2543	}
2544
2545
2546	/**
2547	* Calculates the max data index.
2548	* @returns The number of entries in the paging data array.
2549	*/
2550	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2551	{
2552	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2553	}
2554
2555
2556	/**
2557	* Initializes the paging mode data kept in PGM::paModeData.
2558	*
2559	* @param pVM The VM handle.
2560	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2561	* This is used early in the init process to avoid trouble with PDM
2562	* not being initialized yet.
2563	*/
2564	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2565	{
2566	PPGMMODEDATA pModeData;
2567	int rc;
2568
2569	/*
2570	* Allocate the array on the first call.
2571	*/
2572	if (!pVM->pgm.s.paModeData)
2573	{
2574	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2575	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2576	}
2577
2578	/*
2579	* Initialize the array entries.
2580	*/
2581	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2582	pModeData->uShwType = PGM_TYPE_32BIT;
2583	pModeData->uGstType = PGM_TYPE_REAL;
2584	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2585	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2586	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2587
2588	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2589	pModeData->uShwType = PGM_TYPE_32BIT;
2590	pModeData->uGstType = PGM_TYPE_PROT;
2591	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2592	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2593	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2594
2595	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2596	pModeData->uShwType = PGM_TYPE_32BIT;
2597	pModeData->uGstType = PGM_TYPE_32BIT;
2598	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2599	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2600	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2601
2602	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2603	pModeData->uShwType = PGM_TYPE_PAE;
2604	pModeData->uGstType = PGM_TYPE_REAL;
2605	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2606	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2607	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2608
2609	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2610	pModeData->uShwType = PGM_TYPE_PAE;
2611	pModeData->uGstType = PGM_TYPE_PROT;
2612	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2613	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2614	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2615
2616	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2617	pModeData->uShwType = PGM_TYPE_PAE;
2618	pModeData->uGstType = PGM_TYPE_32BIT;
2619	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2620	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2621	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2622
2623	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2624	pModeData->uShwType = PGM_TYPE_PAE;
2625	pModeData->uGstType = PGM_TYPE_PAE;
2626	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2627	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2628	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2629
2630	#ifdef VBOX_WITH_64_BITS_GUESTS
2631	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2632	pModeData->uShwType = PGM_TYPE_AMD64;
2633	pModeData->uGstType = PGM_TYPE_AMD64;
2634	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2635	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2636	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2637	#endif
2638
2639	/* The nested paging mode. */
2640	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2641	pModeData->uShwType = PGM_TYPE_NESTED;
2642	pModeData->uGstType = PGM_TYPE_REAL;
2643	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2644	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2645
2646	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2647	pModeData->uShwType = PGM_TYPE_NESTED;
2648	pModeData->uGstType = PGM_TYPE_PROT;
2649	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2650	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2651
2652	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2653	pModeData->uShwType = PGM_TYPE_NESTED;
2654	pModeData->uGstType = PGM_TYPE_32BIT;
2655	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2656	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2657
2658	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2659	pModeData->uShwType = PGM_TYPE_NESTED;
2660	pModeData->uGstType = PGM_TYPE_PAE;
2661	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2662	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2663
2664	#ifdef VBOX_WITH_64_BITS_GUESTS
2665	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2666	pModeData->uShwType = PGM_TYPE_NESTED;
2667	pModeData->uGstType = PGM_TYPE_AMD64;
2668	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2669	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2670	#endif
2671
2672	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2673	switch (pVM->pgm.s.enmHostMode)
2674	{
2675	#if HC_ARCH_BITS == 32
2676	case SUPPAGINGMODE_32_BIT:
2677	case SUPPAGINGMODE_32_BIT_GLOBAL:
2678	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2679	{
2680	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2681	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2682	}
2683	# ifdef VBOX_WITH_64_BITS_GUESTS
2684	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2685	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2686	# endif
2687	break;
2688
2689	case SUPPAGINGMODE_PAE:
2690	case SUPPAGINGMODE_PAE_NX:
2691	case SUPPAGINGMODE_PAE_GLOBAL:
2692	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2693	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2694	{
2695	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2696	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2697	}
2698	# ifdef VBOX_WITH_64_BITS_GUESTS
2699	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2700	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2701	# endif
2702	break;
2703	#endif /* HC_ARCH_BITS == 32 */
2704
2705	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
2706	case SUPPAGINGMODE_AMD64:
2707	case SUPPAGINGMODE_AMD64_GLOBAL:
2708	case SUPPAGINGMODE_AMD64_NX:
2709	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2710	# ifdef VBOX_WITH_64_BITS_GUESTS
2711	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
2712	# else
2713	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2714	# endif
2715	{
2716	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2717	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2718	}
2719	break;
2720	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
2721
2722	default:
2723	AssertFailed();
2724	break;
2725	}
2726
2727	/* Extended paging (EPT) / Intel VT-x */
2728	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
2729	pModeData->uShwType = PGM_TYPE_EPT;
2730	pModeData->uGstType = PGM_TYPE_REAL;
2731	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2732	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2733	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2734
2735	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
2736	pModeData->uShwType = PGM_TYPE_EPT;
2737	pModeData->uGstType = PGM_TYPE_PROT;
2738	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2739	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2740	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2741
2742	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
2743	pModeData->uShwType = PGM_TYPE_EPT;
2744	pModeData->uGstType = PGM_TYPE_32BIT;
2745	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2746	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2747	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2748
2749	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
2750	pModeData->uShwType = PGM_TYPE_EPT;
2751	pModeData->uGstType = PGM_TYPE_PAE;
2752	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2753	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2754	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2755
2756	#ifdef VBOX_WITH_64_BITS_GUESTS
2757	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
2758	pModeData->uShwType = PGM_TYPE_EPT;
2759	pModeData->uGstType = PGM_TYPE_AMD64;
2760	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2761	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2762	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2763	#endif
2764	return VINF_SUCCESS;
2765	}
2766
2767
2768	/**
2769	* Switch to different (or relocated in the relocate case) mode data.
2770	*
2771	* @param pVM The VM handle.
2772	* @param pVCpu The VMCPU to operate on.
2773	* @param enmShw The the shadow paging mode.
2774	* @param enmGst The the guest paging mode.
2775	*/
2776	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
2777	{
2778	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
2779
2780	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2781	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2782
2783	/* shadow */
2784	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2785	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2786	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2787	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
2788	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2789
2790	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
2791	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
2792
2793	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2794	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2795
2796
2797	/* guest */
2798	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
2799	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
2800	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
2801	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
2802	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
2803	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
2804	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
2805	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
2806	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
2807	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
2808	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
2809	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
2810
2811	/* both */
2812	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
2813	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
2814	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
2815	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
2816	pVCpu->pgm.s.pfnR3BthSyncPage = pModeData->pfnR3BthSyncPage;
2817	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
2818	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
2819	#ifdef VBOX_STRICT
2820	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
2821	#endif
2822	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
2823	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
2824
2825	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
2826	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
2827	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
2828	pVCpu->pgm.s.pfnRCBthSyncPage = pModeData->pfnRCBthSyncPage;
2829	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
2830	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
2831	#ifdef VBOX_STRICT
2832	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
2833	#endif
2834	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
2835	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
2836
2837	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
2838	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
2839	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
2840	pVCpu->pgm.s.pfnR0BthSyncPage = pModeData->pfnR0BthSyncPage;
2841	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
2842	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
2843	#ifdef VBOX_STRICT
2844	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
2845	#endif
2846	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
2847	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
2848	}
2849
2850
2851	/**
2852	* Calculates the shadow paging mode.
2853	*
2854	* @returns The shadow paging mode.
2855	* @param pVM VM handle.
2856	* @param enmGuestMode The guest mode.
2857	* @param enmHostMode The host mode.
2858	* @param enmShadowMode The current shadow mode.
2859	* @param penmSwitcher Where to store the switcher to use.
2860	* VMMSWITCHER_INVALID means no change.
2861	*/
2862	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
2863	{
2864	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
2865	switch (enmGuestMode)
2866	{
2867	/*
2868	* When switching to real or protected mode we don't change
2869	* anything since it's likely that we'll switch back pretty soon.
2870	*
2871	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
2872	* and is supposed to determine which shadow paging and switcher to
2873	* use during init.
2874	*/
2875	case PGMMODE_REAL:
2876	case PGMMODE_PROTECTED:
2877	if ( enmShadowMode != PGMMODE_INVALID
2878	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
2879	break; /* (no change) */
2880
2881	switch (enmHostMode)
2882	{
2883	case SUPPAGINGMODE_32_BIT:
2884	case SUPPAGINGMODE_32_BIT_GLOBAL:
2885	enmShadowMode = PGMMODE_32_BIT;
2886	enmSwitcher = VMMSWITCHER_32_TO_32;
2887	break;
2888
2889	case SUPPAGINGMODE_PAE:
2890	case SUPPAGINGMODE_PAE_NX:
2891	case SUPPAGINGMODE_PAE_GLOBAL:
2892	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2893	enmShadowMode = PGMMODE_PAE;
2894	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2895	#ifdef DEBUG_bird
2896	if (RTEnvExist("VBOX_32BIT"))
2897	{
2898	enmShadowMode = PGMMODE_32_BIT;
2899	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2900	}
2901	#endif
2902	break;
2903
2904	case SUPPAGINGMODE_AMD64:
2905	case SUPPAGINGMODE_AMD64_GLOBAL:
2906	case SUPPAGINGMODE_AMD64_NX:
2907	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2908	enmShadowMode = PGMMODE_PAE;
2909	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2910	#ifdef DEBUG_bird
2911	if (RTEnvExist("VBOX_32BIT"))
2912	{
2913	enmShadowMode = PGMMODE_32_BIT;
2914	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
2915	}
2916	#endif
2917	break;
2918
2919	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2920	}
2921	break;
2922
2923	case PGMMODE_32_BIT:
2924	switch (enmHostMode)
2925	{
2926	case SUPPAGINGMODE_32_BIT:
2927	case SUPPAGINGMODE_32_BIT_GLOBAL:
2928	enmShadowMode = PGMMODE_32_BIT;
2929	enmSwitcher = VMMSWITCHER_32_TO_32;
2930	break;
2931
2932	case SUPPAGINGMODE_PAE:
2933	case SUPPAGINGMODE_PAE_NX:
2934	case SUPPAGINGMODE_PAE_GLOBAL:
2935	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2936	enmShadowMode = PGMMODE_PAE;
2937	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2938	#ifdef DEBUG_bird
2939	if (RTEnvExist("VBOX_32BIT"))
2940	{
2941	enmShadowMode = PGMMODE_32_BIT;
2942	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2943	}
2944	#endif
2945	break;
2946
2947	case SUPPAGINGMODE_AMD64:
2948	case SUPPAGINGMODE_AMD64_GLOBAL:
2949	case SUPPAGINGMODE_AMD64_NX:
2950	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2951	enmShadowMode = PGMMODE_PAE;
2952	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2953	#ifdef DEBUG_bird
2954	if (RTEnvExist("VBOX_32BIT"))
2955	{
2956	enmShadowMode = PGMMODE_32_BIT;
2957	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
2958	}
2959	#endif
2960	break;
2961
2962	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2963	}
2964	break;
2965
2966	case PGMMODE_PAE:
2967	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
2968	switch (enmHostMode)
2969	{
2970	case SUPPAGINGMODE_32_BIT:
2971	case SUPPAGINGMODE_32_BIT_GLOBAL:
2972	enmShadowMode = PGMMODE_PAE;
2973	enmSwitcher = VMMSWITCHER_32_TO_PAE;
2974	break;
2975
2976	case SUPPAGINGMODE_PAE:
2977	case SUPPAGINGMODE_PAE_NX:
2978	case SUPPAGINGMODE_PAE_GLOBAL:
2979	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2980	enmShadowMode = PGMMODE_PAE;
2981	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2982	break;
2983
2984	case SUPPAGINGMODE_AMD64:
2985	case SUPPAGINGMODE_AMD64_GLOBAL:
2986	case SUPPAGINGMODE_AMD64_NX:
2987	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2988	enmShadowMode = PGMMODE_PAE;
2989	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2990	break;
2991
2992	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2993	}
2994	break;
2995
2996	case PGMMODE_AMD64:
2997	case PGMMODE_AMD64_NX:
2998	switch (enmHostMode)
2999	{
3000	case SUPPAGINGMODE_32_BIT:
3001	case SUPPAGINGMODE_32_BIT_GLOBAL:
3002	enmShadowMode = PGMMODE_AMD64;
3003	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3004	break;
3005
3006	case SUPPAGINGMODE_PAE:
3007	case SUPPAGINGMODE_PAE_NX:
3008	case SUPPAGINGMODE_PAE_GLOBAL:
3009	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3010	enmShadowMode = PGMMODE_AMD64;
3011	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3012	break;
3013
3014	case SUPPAGINGMODE_AMD64:
3015	case SUPPAGINGMODE_AMD64_GLOBAL:
3016	case SUPPAGINGMODE_AMD64_NX:
3017	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3018	enmShadowMode = PGMMODE_AMD64;
3019	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3020	break;
3021
3022	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3023	}
3024	break;
3025
3026
3027	default:
3028	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3029	*penmSwitcher = VMMSWITCHER_INVALID;
3030	return PGMMODE_INVALID;
3031	}
3032	/* Override the shadow mode is nested paging is active. */
3033	if (HWACCMIsNestedPagingActive(pVM))
3034	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3035
3036	*penmSwitcher = enmSwitcher;
3037	return enmShadowMode;
3038	}
3039
3040
3041	/**
3042	* Performs the actual mode change.
3043	* This is called by PGMChangeMode and pgmR3InitPaging().
3044	*
3045	* @returns VBox status code. May suspend or power off the VM on error, but this
3046	* will trigger using FFs and not status codes.
3047	*
3048	* @param pVM VM handle.
3049	* @param pVCpu The VMCPU to operate on.
3050	* @param enmGuestMode The new guest mode. This is assumed to be different from
3051	* the current mode.
3052	*/
3053	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3054	{
3055	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3056	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3057
3058	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3059	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3060
3061	/*
3062	* Calc the shadow mode and switcher.
3063	*/
3064	VMMSWITCHER enmSwitcher;
3065	PGMMODE enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3066	if (enmSwitcher != VMMSWITCHER_INVALID)
3067	{
3068	/*
3069	* Select new switcher.
3070	*/
3071	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3072	if (RT_FAILURE(rc))
3073	{
3074	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3075	return rc;
3076	}
3077	}
3078
3079	/*
3080	* Exit old mode(s).
3081	*/
3082	#if HC_ARCH_BITS == 32
3083	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3084	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3085	&& enmShadowMode == PGMMODE_NESTED);
3086	#else
3087	const bool fForceShwEnterExit = false;
3088	#endif
3089	/* shadow */
3090	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3091	\|\| fForceShwEnterExit)
3092	{
3093	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3094	if (PGM_SHW_PFN(Exit, pVCpu))
3095	{
3096	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3097	if (RT_FAILURE(rc))
3098	{
3099	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3100	return rc;
3101	}
3102	}
3103
3104	}
3105	else
3106	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3107
3108	/* guest */
3109	if (PGM_GST_PFN(Exit, pVCpu))
3110	{
3111	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3112	if (RT_FAILURE(rc))
3113	{
3114	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3115	return rc;
3116	}
3117	}
3118
3119	/*
3120	* Load new paging mode data.
3121	*/
3122	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3123
3124	/*
3125	* Enter new shadow mode (if changed).
3126	*/
3127	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3128	\|\| fForceShwEnterExit)
3129	{
3130	int rc;
3131	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3132	switch (enmShadowMode)
3133	{
3134	case PGMMODE_32_BIT:
3135	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3136	break;
3137	case PGMMODE_PAE:
3138	case PGMMODE_PAE_NX:
3139	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3140	break;
3141	case PGMMODE_AMD64:
3142	case PGMMODE_AMD64_NX:
3143	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3144	break;
3145	case PGMMODE_NESTED:
3146	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3147	break;
3148	case PGMMODE_EPT:
3149	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3150	break;
3151	case PGMMODE_REAL:
3152	case PGMMODE_PROTECTED:
3153	default:
3154	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3155	return VERR_INTERNAL_ERROR;
3156	}
3157	if (RT_FAILURE(rc))
3158	{
3159	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3160	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3161	return rc;
3162	}
3163	}
3164
3165	/*
3166	* Always flag the necessary updates
3167	*/
3168	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3169
3170	/*
3171	* Enter the new guest and shadow+guest modes.
3172	*/
3173	int rc = -1;
3174	int rc2 = -1;
3175	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3176	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3177	switch (enmGuestMode)
3178	{
3179	case PGMMODE_REAL:
3180	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3181	switch (pVCpu->pgm.s.enmShadowMode)
3182	{
3183	case PGMMODE_32_BIT:
3184	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3185	break;
3186	case PGMMODE_PAE:
3187	case PGMMODE_PAE_NX:
3188	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3189	break;
3190	case PGMMODE_NESTED:
3191	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3192	break;
3193	case PGMMODE_EPT:
3194	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3195	break;
3196	case PGMMODE_AMD64:
3197	case PGMMODE_AMD64_NX:
3198	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3199	default: AssertFailed(); break;
3200	}
3201	break;
3202
3203	case PGMMODE_PROTECTED:
3204	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3205	switch (pVCpu->pgm.s.enmShadowMode)
3206	{
3207	case PGMMODE_32_BIT:
3208	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3209	break;
3210	case PGMMODE_PAE:
3211	case PGMMODE_PAE_NX:
3212	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3213	break;
3214	case PGMMODE_NESTED:
3215	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3216	break;
3217	case PGMMODE_EPT:
3218	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3219	break;
3220	case PGMMODE_AMD64:
3221	case PGMMODE_AMD64_NX:
3222	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3223	default: AssertFailed(); break;
3224	}
3225	break;
3226
3227	case PGMMODE_32_BIT:
3228	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3229	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3230	switch (pVCpu->pgm.s.enmShadowMode)
3231	{
3232	case PGMMODE_32_BIT:
3233	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3234	break;
3235	case PGMMODE_PAE:
3236	case PGMMODE_PAE_NX:
3237	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3238	break;
3239	case PGMMODE_NESTED:
3240	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3241	break;
3242	case PGMMODE_EPT:
3243	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3244	break;
3245	case PGMMODE_AMD64:
3246	case PGMMODE_AMD64_NX:
3247	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3248	default: AssertFailed(); break;
3249	}
3250	break;
3251
3252	case PGMMODE_PAE_NX:
3253	case PGMMODE_PAE:
3254	{
3255	uint32_t u32Dummy, u32Features;
3256
3257	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3258	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3259	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3260	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3261
3262	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3263	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3264	switch (pVCpu->pgm.s.enmShadowMode)
3265	{
3266	case PGMMODE_PAE:
3267	case PGMMODE_PAE_NX:
3268	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3269	break;
3270	case PGMMODE_NESTED:
3271	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3272	break;
3273	case PGMMODE_EPT:
3274	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3275	break;
3276	case PGMMODE_32_BIT:
3277	case PGMMODE_AMD64:
3278	case PGMMODE_AMD64_NX:
3279	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3280	default: AssertFailed(); break;
3281	}
3282	break;
3283	}
3284
3285	#ifdef VBOX_WITH_64_BITS_GUESTS
3286	case PGMMODE_AMD64_NX:
3287	case PGMMODE_AMD64:
3288	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3289	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3290	switch (pVCpu->pgm.s.enmShadowMode)
3291	{
3292	case PGMMODE_AMD64:
3293	case PGMMODE_AMD64_NX:
3294	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3295	break;
3296	case PGMMODE_NESTED:
3297	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3298	break;
3299	case PGMMODE_EPT:
3300	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3301	break;
3302	case PGMMODE_32_BIT:
3303	case PGMMODE_PAE:
3304	case PGMMODE_PAE_NX:
3305	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3306	default: AssertFailed(); break;
3307	}
3308	break;
3309	#endif
3310
3311	default:
3312	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3313	rc = VERR_NOT_IMPLEMENTED;
3314	break;
3315	}
3316
3317	/* status codes. */
3318	AssertRC(rc);
3319	AssertRC(rc2);
3320	if (RT_SUCCESS(rc))
3321	{
3322	rc = rc2;
3323	if (RT_SUCCESS(rc)) /* no informational status codes. */
3324	rc = VINF_SUCCESS;
3325	}
3326
3327	/* Notify HWACCM as well. */
3328	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3329	return rc;
3330	}
3331
3332	/**
3333	* Release the pgm lock if owned by the current VCPU
3334	*
3335	* @param pVM The VM to operate on.
3336	*/
3337	VMMR3DECL(void) PGMR3ReleaseOwnedLocks(PVM pVM)
3338	{
3339	while (PDMCritSectIsOwner(&pVM->pgm.s.CritSect))
3340	PDMCritSectLeave(&pVM->pgm.s.CritSect);
3341	}
3342
3343	/**
3344	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3345	*
3346	* @returns VBox status code, fully asserted.
3347	* @param pVM The VM handle.
3348	* @param pVCpu The VMCPU to operate on.
3349	*/
3350	int pgmR3ExitShadowModeBeforePoolFlush(PVM pVM, PVMCPU pVCpu)
3351	{
3352	/* Unmap the old CR3 value before flushing everything. */
3353	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3354	AssertRC(rc);
3355
3356	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3357	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3358	AssertRC(rc);
3359	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3360	return rc;
3361	}
3362
3363
3364	/**
3365	* Called by pgmPoolFlushAllInt after flushing the pool.
3366	*
3367	* @returns VBox status code, fully asserted.
3368	* @param pVM The VM handle.
3369	* @param pVCpu The VMCPU to operate on.
3370	*/
3371	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3372	{
3373	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3374	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3375	Assert(VMCPU_FF_ISSET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3376	AssertRCReturn(rc, rc);
3377	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3378
3379	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3380	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3381	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3382	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3383	return rc;
3384	}
3385
3386
3387	/**
3388	* Dumps a PAE shadow page table.
3389	*
3390	* @returns VBox status code (VINF_SUCCESS).
3391	* @param pVM The VM handle.
3392	* @param pPT Pointer to the page table.
3393	* @param u64Address The virtual address of the page table starts.
3394	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3395	* @param cMaxDepth The maxium depth.
3396	* @param pHlp Pointer to the output functions.
3397	*/
3398	static int pgmR3DumpHierarchyHCPaePT(PVM pVM, PX86PTPAE pPT, uint64_t u64Address, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3399	{
3400	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
3401	{
3402	X86PTEPAE Pte = pPT->a[i];
3403	if (Pte.n.u1Present)
3404	{
3405	pHlp->pfnPrintf(pHlp,
3406	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3407	? "%016llx 3 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n"
3408	: "%08llx 2 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n",
3409	u64Address + ((uint64_t)i << X86_PT_PAE_SHIFT),
3410	Pte.n.u1Write ? 'W' : 'R',
3411	Pte.n.u1User ? 'U' : 'S',
3412	Pte.n.u1Accessed ? 'A' : '-',
3413	Pte.n.u1Dirty ? 'D' : '-',
3414	Pte.n.u1Global ? 'G' : '-',
3415	Pte.n.u1WriteThru ? "WT" : "--",
3416	Pte.n.u1CacheDisable? "CD" : "--",
3417	Pte.n.u1PAT ? "AT" : "--",
3418	Pte.n.u1NoExecute ? "NX" : "--",
3419	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3420	Pte.u & RT_BIT(10) ? '1' : '0',
3421	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED? 'v' : '-',
3422	Pte.u & X86_PTE_PAE_PG_MASK);
3423	}
3424	}
3425	return VINF_SUCCESS;
3426	}
3427
3428
3429	/**
3430	* Dumps a PAE shadow page directory table.
3431	*
3432	* @returns VBox status code (VINF_SUCCESS).
3433	* @param pVM The VM handle.
3434	* @param HCPhys The physical address of the page directory table.
3435	* @param u64Address The virtual address of the page table starts.
3436	* @param cr4 The CR4, PSE is currently used.
3437	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3438	* @param cMaxDepth The maxium depth.
3439	* @param pHlp Pointer to the output functions.
3440	*/
3441	static int pgmR3DumpHierarchyHCPaePD(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3442	{
3443	PX86PDPAE pPD = (PX86PDPAE)MMPagePhys2Page(pVM, HCPhys);
3444	if (!pPD)
3445	{
3446	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory at HCPhys=%RHp was not found in the page pool!\n",
3447	fLongMode ? 16 : 8, u64Address, HCPhys);
3448	return VERR_INVALID_PARAMETER;
3449	}
3450	const bool fBigPagesSupported = fLongMode \|\| !!(cr4 & X86_CR4_PSE);
3451
3452	int rc = VINF_SUCCESS;
3453	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
3454	{
3455	X86PDEPAE Pde = pPD->a[i];
3456	if (Pde.n.u1Present)
3457	{
3458	if (fBigPagesSupported && Pde.b.u1Size)
3459	pHlp->pfnPrintf(pHlp,
3460	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3461	? "%016llx 2 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n"
3462	: "%08llx 1 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n",
3463	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3464	Pde.b.u1Write ? 'W' : 'R',
3465	Pde.b.u1User ? 'U' : 'S',
3466	Pde.b.u1Accessed ? 'A' : '-',
3467	Pde.b.u1Dirty ? 'D' : '-',
3468	Pde.b.u1Global ? 'G' : '-',
3469	Pde.b.u1WriteThru ? "WT" : "--",
3470	Pde.b.u1CacheDisable? "CD" : "--",
3471	Pde.b.u1PAT ? "AT" : "--",
3472	Pde.b.u1NoExecute ? "NX" : "--",
3473	Pde.u & RT_BIT_64(9) ? '1' : '0',
3474	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3475	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3476	Pde.u & X86_PDE_PAE_PG_MASK);
3477	else
3478	{
3479	pHlp->pfnPrintf(pHlp,
3480	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3481	? "%016llx 2 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n"
3482	: "%08llx 1 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n",
3483	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3484	Pde.n.u1Write ? 'W' : 'R',
3485	Pde.n.u1User ? 'U' : 'S',
3486	Pde.n.u1Accessed ? 'A' : '-',
3487	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3488	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3489	Pde.n.u1WriteThru ? "WT" : "--",
3490	Pde.n.u1CacheDisable? "CD" : "--",
3491	Pde.n.u1NoExecute ? "NX" : "--",
3492	Pde.u & RT_BIT_64(9) ? '1' : '0',
3493	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3494	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3495	Pde.u & X86_PDE_PAE_PG_MASK);
3496	if (cMaxDepth >= 1)
3497	{
3498	/** @todo what about using the page pool for mapping PTs? */
3499	uint64_t u64AddressPT = u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT);
3500	RTHCPHYS HCPhysPT = Pde.u & X86_PDE_PAE_PG_MASK;
3501	PX86PTPAE pPT = NULL;
3502	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3503	pPT = (PX86PTPAE)MMPagePhys2Page(pVM, HCPhysPT);
3504	else
3505	{
3506	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3507	{
3508	uint64_t off = u64AddressPT - pMap->GCPtr;
3509	if (off < pMap->cb)
3510	{
3511	const int iPDE = (uint32_t)(off >> X86_PD_SHIFT);
3512	const int iSub = (int)((off >> X86_PD_PAE_SHIFT) & 1); /* MSC is a pain sometimes */
3513	if ((iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0) != HCPhysPT)
3514	pHlp->pfnPrintf(pHlp, "%0*llx error! Mapping error! PT %d has HCPhysPT=%RHp not %RHp is in the PD.\n",
3515	fLongMode ? 16 : 8, u64AddressPT, iPDE,
3516	iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0, HCPhysPT);
3517	pPT = &pMap->aPTs[iPDE].paPaePTsR3[iSub];
3518	}
3519	}
3520	}
3521	int rc2 = VERR_INVALID_PARAMETER;
3522	if (pPT)
3523	rc2 = pgmR3DumpHierarchyHCPaePT(pVM, pPT, u64AddressPT, fLongMode, cMaxDepth - 1, pHlp);
3524	else
3525	pHlp->pfnPrintf(pHlp, "%0*llx error! Page table at HCPhys=%RHp was not found in the page pool!\n",
3526	fLongMode ? 16 : 8, u64AddressPT, HCPhysPT);
3527	if (rc2 < rc && RT_SUCCESS(rc))
3528	rc = rc2;
3529	}
3530	}
3531	}
3532	}
3533	return rc;
3534	}
3535
3536
3537	/**
3538	* Dumps a PAE shadow page directory pointer table.
3539	*
3540	* @returns VBox status code (VINF_SUCCESS).
3541	* @param pVM The VM handle.
3542	* @param HCPhys The physical address of the page directory pointer table.
3543	* @param u64Address The virtual address of the page table starts.
3544	* @param cr4 The CR4, PSE is currently used.
3545	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3546	* @param cMaxDepth The maxium depth.
3547	* @param pHlp Pointer to the output functions.
3548	*/
3549	static int pgmR3DumpHierarchyHCPaePDPT(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3550	{
3551	PX86PDPT pPDPT = (PX86PDPT)MMPagePhys2Page(pVM, HCPhys);
3552	if (!pPDPT)
3553	{
3554	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory pointer table at HCPhys=%RHp was not found in the page pool!\n",
3555	fLongMode ? 16 : 8, u64Address, HCPhys);
3556	return VERR_INVALID_PARAMETER;
3557	}
3558
3559	int rc = VINF_SUCCESS;
3560	const unsigned c = fLongMode ? RT_ELEMENTS(pPDPT->a) : X86_PG_PAE_PDPE_ENTRIES;
3561	for (unsigned i = 0; i < c; i++)
3562	{
3563	X86PDPE Pdpe = pPDPT->a[i];
3564	if (Pdpe.n.u1Present)
3565	{
3566	if (fLongMode)
3567	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3568	"%016llx 1 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3569	u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3570	Pdpe.lm.u1Write ? 'W' : 'R',
3571	Pdpe.lm.u1User ? 'U' : 'S',
3572	Pdpe.lm.u1Accessed ? 'A' : '-',
3573	Pdpe.lm.u3Reserved & 1? '?' : '.', /* ignored */
3574	Pdpe.lm.u3Reserved & 4? '!' : '.', /* mbz */
3575	Pdpe.lm.u1WriteThru ? "WT" : "--",
3576	Pdpe.lm.u1CacheDisable? "CD" : "--",
3577	Pdpe.lm.u3Reserved & 2? "!" : "..",/* mbz */
3578	Pdpe.lm.u1NoExecute ? "NX" : "--",
3579	Pdpe.u & RT_BIT(9) ? '1' : '0',
3580	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3581	Pdpe.u & RT_BIT(11) ? '1' : '0',
3582	Pdpe.u & X86_PDPE_PG_MASK);
3583	else
3584	pHlp->pfnPrintf(pHlp, /P G WT CD AT NX 4M a p ? /
3585	"%08x 0 \| P %c %s %s %s %s .. %c%c%c %016llx\n",
3586	i << X86_PDPT_SHIFT,
3587	Pdpe.n.u4Reserved & 1? '!' : '.', /* mbz */
3588	Pdpe.n.u4Reserved & 4? '!' : '.', /* mbz */
3589	Pdpe.n.u1WriteThru ? "WT" : "--",
3590	Pdpe.n.u1CacheDisable? "CD" : "--",
3591	Pdpe.n.u4Reserved & 2? "!" : "..",/* mbz */
3592	Pdpe.u & RT_BIT(9) ? '1' : '0',
3593	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3594	Pdpe.u & RT_BIT(11) ? '1' : '0',
3595	Pdpe.u & X86_PDPE_PG_MASK);
3596	if (cMaxDepth >= 1)
3597	{
3598	int rc2 = pgmR3DumpHierarchyHCPaePD(pVM, Pdpe.u & X86_PDPE_PG_MASK, u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3599	cr4, fLongMode, cMaxDepth - 1, pHlp);
3600	if (rc2 < rc && RT_SUCCESS(rc))
3601	rc = rc2;
3602	}
3603	}
3604	}
3605	return rc;
3606	}
3607
3608
3609	/**
3610	* Dumps a 32-bit shadow page table.
3611	*
3612	* @returns VBox status code (VINF_SUCCESS).
3613	* @param pVM The VM handle.
3614	* @param HCPhys The physical address of the table.
3615	* @param cr4 The CR4, PSE is currently used.
3616	* @param cMaxDepth The maxium depth.
3617	* @param pHlp Pointer to the output functions.
3618	*/
3619	static int pgmR3DumpHierarchyHcPaePML4(PVM pVM, RTHCPHYS HCPhys, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3620	{
3621	PX86PML4 pPML4 = (PX86PML4)MMPagePhys2Page(pVM, HCPhys);
3622	if (!pPML4)
3623	{
3624	pHlp->pfnPrintf(pHlp, "Page map level 4 at HCPhys=%RHp was not found in the page pool!\n", HCPhys);
3625	return VERR_INVALID_PARAMETER;
3626	}
3627
3628	int rc = VINF_SUCCESS;
3629	for (unsigned i = 0; i < RT_ELEMENTS(pPML4->a); i++)
3630	{
3631	X86PML4E Pml4e = pPML4->a[i];
3632	if (Pml4e.n.u1Present)
3633	{
3634	uint64_t u64Address = ((uint64_t)i << X86_PML4_SHIFT) \| (((uint64_t)i >> (X86_PML4_SHIFT - X86_PDPT_SHIFT - 1)) * 0xffff000000000000ULL);
3635	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3636	"%016llx 0 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3637	u64Address,
3638	Pml4e.n.u1Write ? 'W' : 'R',
3639	Pml4e.n.u1User ? 'U' : 'S',
3640	Pml4e.n.u1Accessed ? 'A' : '-',
3641	Pml4e.n.u3Reserved & 1? '?' : '.', /* ignored */
3642	Pml4e.n.u3Reserved & 4? '!' : '.', /* mbz */
3643	Pml4e.n.u1WriteThru ? "WT" : "--",
3644	Pml4e.n.u1CacheDisable? "CD" : "--",
3645	Pml4e.n.u3Reserved & 2? "!" : "..",/* mbz */
3646	Pml4e.n.u1NoExecute ? "NX" : "--",
3647	Pml4e.u & RT_BIT(9) ? '1' : '0',
3648	Pml4e.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3649	Pml4e.u & RT_BIT(11) ? '1' : '0',
3650	Pml4e.u & X86_PML4E_PG_MASK);
3651
3652	if (cMaxDepth >= 1)
3653	{
3654	int rc2 = pgmR3DumpHierarchyHCPaePDPT(pVM, Pml4e.u & X86_PML4E_PG_MASK, u64Address, cr4, true, cMaxDepth - 1, pHlp);
3655	if (rc2 < rc && RT_SUCCESS(rc))
3656	rc = rc2;
3657	}
3658	}
3659	}
3660	return rc;
3661	}
3662
3663
3664	/**
3665	* Dumps a 32-bit shadow page table.
3666	*
3667	* @returns VBox status code (VINF_SUCCESS).
3668	* @param pVM The VM handle.
3669	* @param pPT Pointer to the page table.
3670	* @param u32Address The virtual address this table starts at.
3671	* @param pHlp Pointer to the output functions.
3672	*/
3673	int pgmR3DumpHierarchyHC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, PCDBGFINFOHLP pHlp)
3674	{
3675	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
3676	{
3677	X86PTE Pte = pPT->a[i];
3678	if (Pte.n.u1Present)
3679	{
3680	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3681	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3682	u32Address + (i << X86_PT_SHIFT),
3683	Pte.n.u1Write ? 'W' : 'R',
3684	Pte.n.u1User ? 'U' : 'S',
3685	Pte.n.u1Accessed ? 'A' : '-',
3686	Pte.n.u1Dirty ? 'D' : '-',
3687	Pte.n.u1Global ? 'G' : '-',
3688	Pte.n.u1WriteThru ? "WT" : "--",
3689	Pte.n.u1CacheDisable? "CD" : "--",
3690	Pte.n.u1PAT ? "AT" : "--",
3691	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3692	Pte.u & RT_BIT(10) ? '1' : '0',
3693	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3694	Pte.u & X86_PDE_PG_MASK);
3695	}
3696	}
3697	return VINF_SUCCESS;
3698	}
3699
3700
3701	/**
3702	* Dumps a 32-bit shadow page directory and page tables.
3703	*
3704	* @returns VBox status code (VINF_SUCCESS).
3705	* @param pVM The VM handle.
3706	* @param cr3 The root of the hierarchy.
3707	* @param cr4 The CR4, PSE is currently used.
3708	* @param cMaxDepth How deep into the hierarchy the dumper should go.
3709	* @param pHlp Pointer to the output functions.
3710	*/
3711	int pgmR3DumpHierarchyHC32BitPD(PVM pVM, uint32_t cr3, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3712	{
3713	PX86PD pPD = (PX86PD)MMPagePhys2Page(pVM, cr3 & X86_CR3_PAGE_MASK);
3714	if (!pPD)
3715	{
3716	pHlp->pfnPrintf(pHlp, "Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK);
3717	return VERR_INVALID_PARAMETER;
3718	}
3719
3720	int rc = VINF_SUCCESS;
3721	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
3722	{
3723	X86PDE Pde = pPD->a[i];
3724	if (Pde.n.u1Present)
3725	{
3726	const uint32_t u32Address = i << X86_PD_SHIFT;
3727	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3728	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3729	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3730	u32Address,
3731	Pde.b.u1Write ? 'W' : 'R',
3732	Pde.b.u1User ? 'U' : 'S',
3733	Pde.b.u1Accessed ? 'A' : '-',
3734	Pde.b.u1Dirty ? 'D' : '-',
3735	Pde.b.u1Global ? 'G' : '-',
3736	Pde.b.u1WriteThru ? "WT" : "--",
3737	Pde.b.u1CacheDisable? "CD" : "--",
3738	Pde.b.u1PAT ? "AT" : "--",
3739	Pde.u & RT_BIT_64(9) ? '1' : '0',
3740	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3741	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3742	Pde.u & X86_PDE4M_PG_MASK);
3743	else
3744	{
3745	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3746	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3747	u32Address,
3748	Pde.n.u1Write ? 'W' : 'R',
3749	Pde.n.u1User ? 'U' : 'S',
3750	Pde.n.u1Accessed ? 'A' : '-',
3751	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3752	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3753	Pde.n.u1WriteThru ? "WT" : "--",
3754	Pde.n.u1CacheDisable? "CD" : "--",
3755	Pde.u & RT_BIT_64(9) ? '1' : '0',
3756	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3757	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3758	Pde.u & X86_PDE_PG_MASK);
3759	if (cMaxDepth >= 1)
3760	{
3761	/** @todo what about using the page pool for mapping PTs? */
3762	RTHCPHYS HCPhys = Pde.u & X86_PDE_PG_MASK;
3763	PX86PT pPT = NULL;
3764	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3765	pPT = (PX86PT)MMPagePhys2Page(pVM, HCPhys);
3766	else
3767	{
3768	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3769	if (u32Address - pMap->GCPtr < pMap->cb)
3770	{
3771	int iPDE = (u32Address - pMap->GCPtr) >> X86_PD_SHIFT;
3772	if (pMap->aPTs[iPDE].HCPhysPT != HCPhys)
3773	pHlp->pfnPrintf(pHlp, "%08x error! Mapping error! PT %d has HCPhysPT=%RHp not %RHp is in the PD.\n",
3774	u32Address, iPDE, pMap->aPTs[iPDE].HCPhysPT, HCPhys);
3775	pPT = pMap->aPTs[iPDE].pPTR3;
3776	}
3777	}
3778	int rc2 = VERR_INVALID_PARAMETER;
3779	if (pPT)
3780	rc2 = pgmR3DumpHierarchyHC32BitPT(pVM, pPT, u32Address, pHlp);
3781	else
3782	pHlp->pfnPrintf(pHlp, "%08x error! Page table at %#x was not found in the page pool!\n", u32Address, HCPhys);
3783	if (rc2 < rc && RT_SUCCESS(rc))
3784	rc = rc2;
3785	}
3786	}
3787	}
3788	}
3789
3790	return rc;
3791	}
3792
3793
3794	/**
3795	* Dumps a 32-bit shadow page table.
3796	*
3797	* @returns VBox status code (VINF_SUCCESS).
3798	* @param pVM The VM handle.
3799	* @param pPT Pointer to the page table.
3800	* @param u32Address The virtual address this table starts at.
3801	* @param PhysSearch Address to search for.
3802	*/
3803	int pgmR3DumpHierarchyGC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, RTGCPHYS PhysSearch)
3804	{
3805	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
3806	{
3807	X86PTE Pte = pPT->a[i];
3808	if (Pte.n.u1Present)
3809	{
3810	Log(( /P R S A D G WT CD AT NX 4M a m d /
3811	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3812	u32Address + (i << X86_PT_SHIFT),
3813	Pte.n.u1Write ? 'W' : 'R',
3814	Pte.n.u1User ? 'U' : 'S',
3815	Pte.n.u1Accessed ? 'A' : '-',
3816	Pte.n.u1Dirty ? 'D' : '-',
3817	Pte.n.u1Global ? 'G' : '-',
3818	Pte.n.u1WriteThru ? "WT" : "--",
3819	Pte.n.u1CacheDisable? "CD" : "--",
3820	Pte.n.u1PAT ? "AT" : "--",
3821	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3822	Pte.u & RT_BIT(10) ? '1' : '0',
3823	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3824	Pte.u & X86_PDE_PG_MASK));
3825
3826	if ((Pte.u & X86_PDE_PG_MASK) == PhysSearch)
3827	{
3828	uint64_t fPageShw = 0;
3829	RTHCPHYS pPhysHC = 0;
3830
3831	/** @todo SMP support!! */
3832	PGMShwGetPage(&pVM->aCpus[0], (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), &fPageShw, &pPhysHC);
3833	Log(("Found %RGp at %RGv -> flags=%llx\n", PhysSearch, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), fPageShw));
3834	}
3835	}
3836	}
3837	return VINF_SUCCESS;
3838	}
3839
3840
3841	/**
3842	* Dumps a 32-bit guest page directory and page tables.
3843	*
3844	* @returns VBox status code (VINF_SUCCESS).
3845	* @param pVM The VM handle.
3846	* @param cr3 The root of the hierarchy.
3847	* @param cr4 The CR4, PSE is currently used.
3848	* @param PhysSearch Address to search for.
3849	*/
3850	VMMR3DECL(int) PGMR3DumpHierarchyGC(PVM pVM, uint64_t cr3, uint64_t cr4, RTGCPHYS PhysSearch)
3851	{
3852	bool fLongMode = false;
3853	const unsigned cch = fLongMode ? 16 : 8; NOREF(cch);
3854	PX86PD pPD = 0;
3855
3856	int rc = PGM_GCPHYS_2_PTR(pVM, cr3 & X86_CR3_PAGE_MASK, &pPD);
3857	if (RT_FAILURE(rc) \|\| !pPD)
3858	{
3859	Log(("Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK));
3860	return VERR_INVALID_PARAMETER;
3861	}
3862
3863	Log(("cr3=%08x cr4=%08x%s\n"
3864	"%-*s P - Present\n"
3865	"%-*s \| R/W - Read (0) / Write (1)\n"
3866	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3867	"%-*s \| \| \| A - Accessed\n"
3868	"%-*s \| \| \| \| D - Dirty\n"
3869	"%-*s \| \| \| \| \| G - Global\n"
3870	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3871	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3872	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3873	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3874	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3875	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3876	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3877	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3878	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3879	- W U - - - -- -- -- -- -- 010 */
3880	, cr3, cr4, fLongMode ? " Long Mode" : "",
3881	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3882	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address"));
3883
3884	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
3885	{
3886	X86PDE Pde = pPD->a[i];
3887	if (Pde.n.u1Present)
3888	{
3889	const uint32_t u32Address = i << X86_PD_SHIFT;
3890
3891	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3892	Log(( /P R S A D G WT CD AT NX 4M a m d /
3893	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3894	u32Address,
3895	Pde.b.u1Write ? 'W' : 'R',
3896	Pde.b.u1User ? 'U' : 'S',
3897	Pde.b.u1Accessed ? 'A' : '-',
3898	Pde.b.u1Dirty ? 'D' : '-',
3899	Pde.b.u1Global ? 'G' : '-',
3900	Pde.b.u1WriteThru ? "WT" : "--",
3901	Pde.b.u1CacheDisable? "CD" : "--",
3902	Pde.b.u1PAT ? "AT" : "--",
3903	Pde.u & RT_BIT(9) ? '1' : '0',
3904	Pde.u & RT_BIT(10) ? '1' : '0',
3905	Pde.u & RT_BIT(11) ? '1' : '0',
3906	pgmGstGet4MBPhysPage(&pVM->pgm.s, Pde)));
3907	/** @todo PhysSearch */
3908	else
3909	{
3910	Log(( /P R S A D G WT CD AT NX 4M a m d /
3911	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3912	u32Address,
3913	Pde.n.u1Write ? 'W' : 'R',
3914	Pde.n.u1User ? 'U' : 'S',
3915	Pde.n.u1Accessed ? 'A' : '-',
3916	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3917	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3918	Pde.n.u1WriteThru ? "WT" : "--",
3919	Pde.n.u1CacheDisable? "CD" : "--",
3920	Pde.u & RT_BIT(9) ? '1' : '0',
3921	Pde.u & RT_BIT(10) ? '1' : '0',
3922	Pde.u & RT_BIT(11) ? '1' : '0',
3923	Pde.u & X86_PDE_PG_MASK));
3924	////if (cMaxDepth >= 1)
3925	{
3926	/** @todo what about using the page pool for mapping PTs? */
3927	RTGCPHYS GCPhys = Pde.u & X86_PDE_PG_MASK;
3928	PX86PT pPT = NULL;
3929
3930	rc = PGM_GCPHYS_2_PTR(pVM, GCPhys, &pPT);
3931
3932	int rc2 = VERR_INVALID_PARAMETER;
3933	if (pPT)
3934	rc2 = pgmR3DumpHierarchyGC32BitPT(pVM, pPT, u32Address, PhysSearch);
3935	else
3936	Log(("%08x error! Page table at %#x was not found in the page pool!\n", u32Address, GCPhys));
3937	if (rc2 < rc && RT_SUCCESS(rc))
3938	rc = rc2;
3939	}
3940	}
3941	}
3942	}
3943
3944	return rc;
3945	}
3946
3947
3948	/**
3949	* Dumps a page table hierarchy use only physical addresses and cr4/lm flags.
3950	*
3951	* @returns VBox status code (VINF_SUCCESS).
3952	* @param pVM The VM handle.
3953	* @param cr3 The root of the hierarchy.
3954	* @param cr4 The cr4, only PAE and PSE is currently used.
3955	* @param fLongMode Set if long mode, false if not long mode.
3956	* @param cMaxDepth Number of levels to dump.
3957	* @param pHlp Pointer to the output functions.
3958	*/
3959	VMMR3DECL(int) PGMR3DumpHierarchyHC(PVM pVM, uint64_t cr3, uint64_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3960	{
3961	if (!pHlp)
3962	pHlp = DBGFR3InfoLogHlp();
3963	if (!cMaxDepth)
3964	return VINF_SUCCESS;
3965	const unsigned cch = fLongMode ? 16 : 8;
3966	pHlp->pfnPrintf(pHlp,
3967	"cr3=%08x cr4=%08x%s\n"
3968	"%-*s P - Present\n"
3969	"%-*s \| R/W - Read (0) / Write (1)\n"
3970	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3971	"%-*s \| \| \| A - Accessed\n"
3972	"%-*s \| \| \| \| D - Dirty\n"
3973	"%-*s \| \| \| \| \| G - Global\n"
3974	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3975	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3976	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3977	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3978	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3979	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3980	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3981	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3982	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3983	- W U - - - -- -- -- -- -- 010 */
3984	, cr3, cr4, fLongMode ? " Long Mode" : "",
3985	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3986	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address");
3987	if (cr4 & X86_CR4_PAE)
3988	{
3989	if (fLongMode)
3990	return pgmR3DumpHierarchyHcPaePML4(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3991	return pgmR3DumpHierarchyHCPaePDPT(pVM, cr3 & X86_CR3_PAE_PAGE_MASK, 0, cr4, false, cMaxDepth, pHlp);
3992	}
3993	return pgmR3DumpHierarchyHC32BitPD(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3994	}
3995
3996	#ifdef VBOX_WITH_DEBUGGER
3997
3998	/**
3999	* The '.pgmram' command.
4000	*
4001	* @returns VBox status.
4002	* @param pCmd Pointer to the command descriptor (as registered).
4003	* @param pCmdHlp Pointer to command helper functions.
4004	* @param pVM Pointer to the current VM (if any).
4005	* @param paArgs Pointer to (readonly) array of arguments.
4006	* @param cArgs Number of arguments in the array.
4007	*/
4008	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4009	{
4010	/*
4011	* Validate input.
4012	*/
4013	if (!pVM)
4014	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4015	if (!pVM->pgm.s.pRamRangesRC)
4016	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
4017
4018	/*
4019	* Dump the ranges.
4020	*/
4021	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
4022	PPGMRAMRANGE pRam;
4023	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
4024	{
4025	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
4026	"%RGp - %RGp %p\n",
4027	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
4028	if (RT_FAILURE(rc))
4029	return rc;
4030	}
4031
4032	return VINF_SUCCESS;
4033	}
4034
4035
4036	/**
4037	* The '.pgmerror' and '.pgmerroroff' commands.
4038	*
4039	* @returns VBox status.
4040	* @param pCmd Pointer to the command descriptor (as registered).
4041	* @param pCmdHlp Pointer to command helper functions.
4042	* @param pVM Pointer to the current VM (if any).
4043	* @param paArgs Pointer to (readonly) array of arguments.
4044	* @param cArgs Number of arguments in the array.
4045	*/
4046	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4047	{
4048	/*
4049	* Validate input.
4050	*/
4051	if (!pVM)
4052	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4053	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
4054	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
4055
4056	if (!cArgs)
4057	{
4058	/*
4059	* Print the list of error injection locations with status.
4060	*/
4061	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
4062	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
4063	}
4064	else
4065	{
4066
4067	/*
4068	* String switch on where to inject the error.
4069	*/
4070	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
4071	const char *pszWhere = paArgs[0].u.pszString;
4072	if (!strcmp(pszWhere, "handy"))
4073	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
4074	else
4075	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
4076	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
4077	}
4078	return VINF_SUCCESS;
4079	}
4080
4081
4082	/**
4083	* The '.pgmsync' command.
4084	*
4085	* @returns VBox status.
4086	* @param pCmd Pointer to the command descriptor (as registered).
4087	* @param pCmdHlp Pointer to command helper functions.
4088	* @param pVM Pointer to the current VM (if any).
4089	* @param paArgs Pointer to (readonly) array of arguments.
4090	* @param cArgs Number of arguments in the array.
4091	*/
4092	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4093	{
4094	/** @todo SMP support */
4095	PVMCPU pVCpu = &pVM->aCpus[0];
4096
4097	/*
4098	* Validate input.
4099	*/
4100	if (!pVM)
4101	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4102
4103	/*
4104	* Force page directory sync.
4105	*/
4106	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
4107
4108	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
4109	if (RT_FAILURE(rc))
4110	return rc;
4111
4112	return VINF_SUCCESS;
4113	}
4114
4115
4116	#ifdef VBOX_STRICT
4117	/**
4118	* The '.pgmassertcr3' command.
4119	*
4120	* @returns VBox status.
4121	* @param pCmd Pointer to the command descriptor (as registered).
4122	* @param pCmdHlp Pointer to command helper functions.
4123	* @param pVM Pointer to the current VM (if any).
4124	* @param paArgs Pointer to (readonly) array of arguments.
4125	* @param cArgs Number of arguments in the array.
4126	*/
4127	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4128	{
4129	/** @todo SMP support!! */
4130	PVMCPU pVCpu = &pVM->aCpus[0];
4131
4132	/*
4133	* Validate input.
4134	*/
4135	if (!pVM)
4136	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4137
4138	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
4139	if (RT_FAILURE(rc))
4140	return rc;
4141
4142	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
4143
4144	return VINF_SUCCESS;
4145	}
4146	#endif /* VBOX_STRICT */
4147
4148
4149	/**
4150	* The '.pgmsyncalways' command.
4151	*
4152	* @returns VBox status.
4153	* @param pCmd Pointer to the command descriptor (as registered).
4154	* @param pCmdHlp Pointer to command helper functions.
4155	* @param pVM Pointer to the current VM (if any).
4156	* @param paArgs Pointer to (readonly) array of arguments.
4157	* @param cArgs Number of arguments in the array.
4158	*/
4159	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4160	{
4161	/** @todo SMP support!! */
4162	PVMCPU pVCpu = &pVM->aCpus[0];
4163
4164	/*
4165	* Validate input.
4166	*/
4167	if (!pVM)
4168	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4169
4170	/*
4171	* Force page directory sync.
4172	*/
4173	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
4174	{
4175	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
4176	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
4177	}
4178	else
4179	{
4180	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
4181	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
4182	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
4183	}
4184	}
4185
4186
4187	/**
4188	* The '.pgmsyncalways' command.
4189	*
4190	* @returns VBox status.
4191	* @param pCmd Pointer to the command descriptor (as registered).
4192	* @param pCmdHlp Pointer to command helper functions.
4193	* @param pVM Pointer to the current VM (if any).
4194	* @param paArgs Pointer to (readonly) array of arguments.
4195	* @param cArgs Number of arguments in the array.
4196	*/
4197	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4198	{
4199	/*
4200	* Validate input.
4201	*/
4202	if (!pVM)
4203	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4204	if ( cArgs < 1
4205	\|\| cArgs > 2
4206	\|\| paArgs[0].enmType != DBGCVAR_TYPE_STRING
4207	\|\| ( cArgs > 1
4208	&& paArgs[1].enmType != DBGCVAR_TYPE_STRING))
4209	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: parser error, invalid arguments.\n");
4210	if ( cArgs >= 2
4211	&& strcmp(paArgs[1].u.pszString, "nozero"))
4212	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
4213	bool fIncZeroPgs = cArgs < 2;
4214
4215	/*
4216	* Open the output file and get the ram parameters.
4217	*/
4218	RTFILE hFile;
4219	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
4220	if (RT_FAILURE(rc))
4221	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
4222
4223	uint32_t cbRamHole = 0;
4224	CFGMR3QueryU32Def(CFGMR3GetRoot(pVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
4225	uint64_t cbRam = 0;
4226	CFGMR3QueryU64Def(CFGMR3GetRoot(pVM), "RamSize", &cbRam, 0);
4227	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
4228
4229	/*
4230	* Dump the physical memory, page by page.
4231	*/
4232	RTGCPHYS GCPhys = 0;
4233	char abZeroPg[PAGE_SIZE];
4234	RT_ZERO(abZeroPg);
4235
4236	pgmLock(pVM);
4237	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesR3;
4238	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
4239	pRam = pRam->pNextR3)
4240	{
4241	/* fill the gap */
4242	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
4243	{
4244	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
4245	{
4246	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
4247	GCPhys += PAGE_SIZE;
4248	}
4249	}
4250
4251	PCPGMPAGE pPage = &pRam->aPages[0];
4252	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
4253	{
4254	if (PGM_PAGE_IS_ZERO(pPage))
4255	{
4256	if (fIncZeroPgs)
4257	{
4258	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
4259	if (RT_FAILURE(rc))
4260	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
4261	}
4262	}
4263	else
4264	{
4265	switch (PGM_PAGE_GET_TYPE(pPage))
4266	{
4267	case PGMPAGETYPE_RAM:
4268	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
4269	case PGMPAGETYPE_ROM:
4270	case PGMPAGETYPE_MMIO2:
4271	{
4272	void const *pvPage;
4273	PGMPAGEMAPLOCK Lock;
4274	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
4275	if (RT_SUCCESS(rc))
4276	{
4277	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
4278	PGMPhysReleasePageMappingLock(pVM, &Lock);
4279	if (RT_FAILURE(rc))
4280	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
4281	}
4282	else
4283	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
4284	break;
4285	}
4286
4287	default:
4288	AssertFailed();
4289	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
4290	case PGMPAGETYPE_MMIO:
4291	if (fIncZeroPgs)
4292	{
4293	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
4294	if (RT_FAILURE(rc))
4295	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
4296	}
4297	break;
4298	}
4299	}
4300
4301
4302	/* advance */
4303	GCPhys += PAGE_SIZE;
4304	pPage++;
4305	}
4306	}
4307	pgmUnlock(pVM);
4308
4309	RTFileClose(hFile);
4310	if (RT_SUCCESS(rc))
4311	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
4312	return VINF_SUCCESS;
4313	}
4314
4315	#endif /* VBOX_WITH_DEBUGGER */
4316
4317	/**
4318	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
4319	*/
4320	typedef struct PGMCHECKINTARGS
4321	{
4322	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
4323	PPGMPHYSHANDLER pPrevPhys;
4324	PPGMVIRTHANDLER pPrevVirt;
4325	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
4326	PVM pVM;
4327	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
4328
4329	/**
4330	* Validate a node in the physical handler tree.
4331	*
4332	* @returns 0 on if ok, other wise 1.
4333	* @param pNode The handler node.
4334	* @param pvUser pVM.
4335	*/
4336	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4337	{
4338	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4339	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
4340	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4341	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4342	AssertReleaseMsg( !pArgs->pPrevPhys
4343	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
4344	("pPrevPhys=%p %RGp-%RGp %s\n"
4345	" pCur=%p %RGp-%RGp %s\n",
4346	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
4347	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4348	pArgs->pPrevPhys = pCur;
4349	return 0;
4350	}
4351
4352
4353	/**
4354	* Validate a node in the virtual handler tree.
4355	*
4356	* @returns 0 on if ok, other wise 1.
4357	* @param pNode The handler node.
4358	* @param pvUser pVM.
4359	*/
4360	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
4361	{
4362	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4363	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
4364	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4365	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4366	AssertReleaseMsg( !pArgs->pPrevVirt
4367	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
4368	("pPrevVirt=%p %RGv-%RGv %s\n"
4369	" pCur=%p %RGv-%RGv %s\n",
4370	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
4371	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4372	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
4373	{
4374	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
4375	("pCur=%p %RGv-%RGv %s\n"
4376	"iPage=%d offVirtHandle=%#x expected %#x\n",
4377	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
4378	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
4379	}
4380	pArgs->pPrevVirt = pCur;
4381	return 0;
4382	}
4383
4384
4385	/**
4386	* Validate a node in the virtual handler tree.
4387	*
4388	* @returns 0 on if ok, other wise 1.
4389	* @param pNode The handler node.
4390	* @param pvUser pVM.
4391	*/
4392	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4393	{
4394	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4395	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
4396	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
4397	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
4398	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
4399	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4400	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4401	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4402	" pCur=%p %RGp-%RGp\n",
4403	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4404	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4405	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4406	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4407	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4408	" pCur=%p %RGp-%RGp\n",
4409	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4410	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4411	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
4412	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4413	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4414	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
4415	{
4416	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
4417	for (;;)
4418	{
4419	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4420	AssertReleaseMsg(pCur2 != pCur,
4421	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4422	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4423	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4424	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4425	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4426	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4427	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4428	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4429	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4430	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4431	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4432	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4433	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4434	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4435	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4436	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4437	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4438	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4439	break;
4440	}
4441	}
4442
4443	pArgs->pPrevPhys2Virt = pCur;
4444	return 0;
4445	}
4446
4447
4448	/**
4449	* Perform an integrity check on the PGM component.
4450	*
4451	* @returns VINF_SUCCESS if everything is fine.
4452	* @returns VBox error status after asserting on integrity breach.
4453	* @param pVM The VM handle.
4454	*/
4455	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4456	{
4457	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4458
4459	/*
4460	* Check the trees.
4461	*/
4462	int cErrors = 0;
4463	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4464	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4465	PGMCHECKINTARGS Args = s_LeftToRight;
4466	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4467	Args = s_RightToLeft;
4468	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4469	Args = s_LeftToRight;
4470	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4471	Args = s_RightToLeft;
4472	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4473	Args = s_LeftToRight;
4474	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4475	Args = s_RightToLeft;
4476	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4477	Args = s_LeftToRight;
4478	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4479	Args = s_RightToLeft;
4480	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4481
4482	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4483	}
4484
4485

注意: 瀏覽 TracBrowser 來幫助您使用儲存庫瀏覽器

source: vbox/trunk/src/VBox/VMM/PGM.cpp@ 26112

以其他格式下載: