Dosimetric Evaluation of a Treatment Planning System Using the Aapm Medical Physics Practice Guideline 5.a (MPPG 5.a) Validation Tests

In this work, the AAPM Medical Physics Practice Guideline 5.a (MPPG 5.a) validation tests package was used to evaluate the dosimetric performance of a new version of the Eclipse treatment planning system (TPS) algorithms. A series of tests were developed and comparisons between TPS calculated and corresponding beam data measurements have been performed for basic beam validation, heterogeneity correction and IMRT/VMAT dose validation tests. Measurements were performed using a Varian IX Linear Accelerator with the 6 MV, 6MV FFF and 18 MV photon beams, and 6, 9, 12, 16, 20 MeV electron beams. Results for basic dose validation tests yielded differences within 3% for all point doses and pass rate greater than 95% for all depth proles using 3%/3mm criteria. For testing the ability of the TPS in accounting for tissue inhomogeneity, corresponding comparisons were performed with the presence of a heterogeneous media to simulate an air inhomogeneity. Results showed a mean deviation between the TPS calculated and measured of 1.9%, reaching a maximum of 2.8% for the AAA algorithm. For IMRT/VMAT validation tests, our local criteria passing rate of 95% was used, but no consensus of the tolerance exists. Our results agree well with the data reported accuracy in previous studies of Eclipse TPS. In summary, the AAPM MPPG 5.a validation tests are a valuable package for evaluating dose calculation accuracy and are very useful for TPS upgrade checks, commissioning tests and routine TPS QA. MPPG 5.a. The results obtained demonstrated the accuracy of the Eclipse version 15.6 TPS algorithms, AAA, AXB and eMC, for photon and electron beams. Basic validation, heterogeneity corrections and IMRT/VMAT dose validation for Eclipse’s photon and electron algorithms have been demonstrated. The MPPG 5.a validation tests provides the opportunity for the physicist to understand the accuracy and limitations of the dose algorithm commissioned prior to clinical implementation. Comparisons between TPS calculated beam proles and point doses were performed for three sections. An extensive set of measurements were also performed, which were compared to corresponding TPS calculations. The data collected from the dosimetric performance results obtained during the validations testing is helpful for TPS upgrade or commissioning and for routine TPS QA.

Introduction A treatment planning system (TPS) is considered an essential component of external beam radiation therapy. It is used to generate beam shapes and dose distributions with the intent to maximize tumour dose and minimize normal tissue complications. To achieve this goal, a dose coverage to a tumour volume should be within the +7% to -5% range [1]. In general, treatment planning is a complex process that involves many steps and relies on many factors to achieve the required accuracy, and therefore, a standard and stringent Quality Assurance (QA) program of the TPS has to be implemented. Several guidance documents were published to help medical physicists in developing tests for commissioning and validation of TPS [2][3][4]. The AAPM TG-53 [2] and the IAEA TECDOC-430 published comprehensive reports on commissioning and QA of a computerized TPS for radiotherapy [3]. Recently, the AAPM published the Medical Physics Practice Guideline 5.a (MPPG 5.a), which provides concise guidelines on the commissioning and QA of TPS dose calculations algorithms [5]. Although these previous reports can be considered as complete [2][3][4] and are strongly encouraged for the implementation of a robust and comprehensive QA program, their drawback is associated with the need for lots of resources and time to be implemented effectively which can be di cult to achieve in a limited-resources department. Even though the MPPG 5.a sets minimum requirements for commissioning and QA of TPS dose calculations, it has also offered the clinical physicists the exibility in data acquisition using a range of detectors and tools. The validation process described by the MPPG5.a is divided into four sections; basic photon beams, photon beam heterogeneity corrections, IMRT/VMAT validation, and electron beam validation.
These are sections 5 to 8, respectively, of the report [5]. Although many groups have published their experience in commissioning and verifying different TPS algorithms [6-9], a few references are found on implementation of the MPPG 5.a validation tests [10,11]. In the present work, the evaluation of the accuracy of Eclipse TPS (Varian Medical Systems, Palo Alto, CA) calculation algorithms is performed comparing the measured values with the calculated one. The MPPG5.a validation tests are implemented with different organization of the original validation suits of validation tests and are made as follows: A) the basic photon and electron tests are merged into one section. The heterogeneity corrections for both photon and electron are also combined into one section (B); and Section C) is IMRT/VMAT dose validation tests. The goal of this work was to evaluate the dosimetric accuracy of the newer version of the Eclipse TPS dose calculation algorithms. To this aim, the AAPM MPPG 5.a validation test packages has been used.

A Treatment planning System
The Eclipse TPS Version 15.6.05, was used for all calculations in this study. The dose calculation algorithms employed by the TPS were Anisotropic Analytical Algorithm (AAA) and Acuros XB algorithm (AXB) for photon beam and the Electron Monte Carlo (eMC) for electron beam. Dose calculations were performed using a virtual water tank (a cube of water created in the Eclipse TPS) for most basic validation tests. For heterogeneity correction tests, plans were made in a heterogonous phantom in TPS, and a number of points were added to the plan representing the location of the ionization chamber. For IMRT/VMAT dose validation tests, clinical reference plans were re-calculated in ArcCHECK's virtual phantom and exported to the machine for delivery. All plans were generated in the TPS for the MPPG 5.a validation tests in a single test patient. For basic validation tests, each plan contained beams with the same eld size but different energies. All dose calculations were performed using a grid size of a 2 mm and exported separately for each beam for comparisons.

B Measurement equipment and technique
Beam data and point doses were acquired following the MPPG 5.a's recommendations using various detectors in a scanning water tank, solid water phantoms, or a 3D detector array device. Table 1 summarises the MPPG 5.a tests with an associated description alongside the equipment used. The radiation source was a Varian iX linear accelerator (Linac) equipped with a Millennium 120 multi-leaf collimator (MLC). The Linac delivers photon beams of 6 and 18 MV (with attening lter), 6 FFF ( attening lter-free), and electron beams of 6, 9, 12, 16 and 20 MeV. A PTW 3D MP3-M water tank (PTW, Freiburg, Germany) was used with the following PTW detectors; Semi ex ionisation chamber (active volume of 0.125 cm 3 ), a 3D Pinpoint ionization chamber (active volume of 0.016 cm 3 ), and an Electron diode (active volume of 0.03 cm 3 ) for dosimetric measurements with the acquired data processed using PTW's MEPHYSTO software version 3.4 (PTW, Freiburg, Germany). Following the TRS-398 protocol for reference dosimetry [12], a PTW Farmer and Roos Parallel Plate chambers were used with a PTW UNIDOS electrometer. For heterogeneity measurements, custom made phantoms were designed from solid water slabs (Gammex-RMI, Middleton, WI, USA) and Styrofoam to mimic lung tissue as shown in Fig 2. A schematic of phantoms dimensions, all with 10 cm backscatter after the Styrofoam is illustrated Fig 3; (a) photon heterogeneity dose measurements, (b) 6 MeV electron dose measurements; (c) 9 MeV electron dose measurements; and (d) for 12-20 MeV electron dose measurements. Furthermore, a Gammex electron density CT phantom (Gammex RMI, Middelton, WI, USA) was also used to establish the CT-to-ED curve, and the Sun Nuclear 3D ArcCHECK diode array (Sun Nuclear, Melbourne, FL, USA) was used for IMRT/VMAT veri cation. All phantoms were scanned on a Phillips Big Bore computed tomography (CT) scanner (Philips Healthcare, Amsterdam, Netherlands) with a 2-mm slice thickness, and then CT datasets were exported to the TPS for dose calculations 2. C The MPPG 5.a validation tests

C. 1 Basic beam validation
The basic photon and electron beam algorithm validation tests 5.1 -5.3 are essential tests to verify the agreement of the dose distribution in the planning module with the commissioning data, reference clinical calibration conditions, beam pro les and output factors for large and small eld sizes. In test 5.1, dose distributions of a 40 x 40 cm 2 eld incident on a large water phantom at an SSD of 100 cm were compared to the corresponding commissioning pro les. Test 5.2; photon and electron absolute dose calculation, under reference conditions, were also performed; a eld size of 10 x10 cm 2 at an SSD of 100 cm, at reference depth were compared to the measured dose per MU at the same reference conditions. Test 5.3, measured percentage depth doses (PDD), beam pro les, and output factors for square eld sizes of 2, 3, 4, 5, 7, 10, 20, and 40 cm 2 were compared to calculated data. Furthermore, the MPPG 5.a recommends six additional basic validation tests for photon beams, ve with static MLC eld shapes with the jaws placed at clinically relevant positions (5.4-5.8) and one with a nonphysical wedge (5.9). For electron beams, two validation tests, 8.1 and 8.2, were performed in a homogenous water phantom. Test 8.1 used a custom cut-out shape in a 15 x 15 applicator. This eld represents a clinically relevant shaped eld. Test 8.2 aims to verify the surface irregularities obliquity using a standard 10 x 10 applicator at an oblique beam incidence of 20°. Fig 1 illustrates the Beam's Eye View of the MLC elds created in the TPS. Test 5.4, a small MLC-shape, was intended to mimic a small MLC de ned treatment eld. A large MLC-shaped eld with extensive blocking (e.g., mantle shape) was used in test 5.5. Test 5.6 is an off-axis eld with the maximum allowed leaf over-travel in the X-direction with the Y direction de ned by the jaws. Test 5.7 is an irregular MLC eld shape. Test 5.8 is an MLC de ned eld at an oblique incident angle of 20° from the vertical. Test 5.9 veri es a large eld with an enhanced dynamic wedge (EDW) measured using an ArcCHECK diode array. A 20 × 20 cm 2 EDW eld with wedge angles of 15 º, 30 º, 45 º and 60º were measured for 6 and 18 MV photon beams. For this test, two sets of measurements were performed, one with Y1-IN wedge orientation and the other with Y2-OUT wedge orientation.
All beam pro le and PDD measurements for basic validation tests (5.4-5.9, 8.1-8.2) were obtained in a 3D scanning water tank. For photons, one PDD and two beam pro les at dmax, 10 cm or 25 cm depth were measured in the inplane and crossplane directions. For electrons, one PDD and one inplane and one crossplane pro les were measured at a depth of the 90% isodose. All scanning beam data were compared to the calculated data from Eclipse TPS using an open-source scanning data analysis tool ScanDoseMatch (http://www.qxrayconsulting.com/sdm/) [7,13]. Gamma analysis was performed using a gamma criterion of a 2% / 2 mm following the MPPG 5.a recommendations. Besides the scanned data, additional veri cation point doses were also acquired at 1.5 cm and 10 cm depths along the central axis (CAX) for tests 5.4, 5.5, 5.7 and 5.8. The absolute dose at each point of the measured pro le was acquired using a calibrated Pinpoint chamber.

C. 2 Heterogeneity correction validation
The minimum recommended validation tests include con rming the CT-to-ED curve (test 6.1) and verifying TPS calculations beyond heterogeneities (tests 6.2 & 8.3). The CT-to-ED curve was veri ed in the TPS by scanning an electron density phantom and con rming the CT values of various inserts were reported correctly in the TPS. For dose veri cation beyond the heterogeneous media for photon beams (test 6.2), measurements were taken at two points located along the CAX; one upper and one lower placed at 4 and 22 cm (4 cm behind the Styrofoam slab), respectively. For electron beams (test 8.3), one measurement was collected close to dmax in heterogeneity phantom, and at least two measurements were acquired along the CAX in the dose fall-off region. Measurements near dmax were taken at 2.5 cm and 3.5 cm for 6 and 9 MeV and at 6.5 cm for 12, 16 and 20 MeV. Measurements along the dose fall-off region were taken at 3 and 3.5 cm for 6 MeV, 4 and 4.5 cm for 9 MeV, 9 and10 cm for 12 MeV, 7 and 10 cm for 16 MeV, and 7 and 11 cm for 20 MeV. All measurements were compared to the Eclipse TPS points representing the location of the ionization chamber.

C. 3 IMRT/VMAT dose validation
Several validation tests are recommended for IMRT/VMAT delivery modality. Tests 7.1 and 7.2 are for a very small MLC de ned shape eld. The PDD and output factor for a 2 x 2 cm 2 MLC-shape eld were measured in a scanning water tank with a PTW E Diode. For tests 7.3 and 7.4, twelve cases; two IMRT and ten VMAT plans for prostate, endometrium, pelvis, head and neck, brain, and rectum treatment sites, were used. All plans were re-calculated onto the ArcCHECK veri cation phantom, delivered and evaluated using gamma criteria of 3% /3 mm and 2%/2 mm with absolute dose, global normalization, and a low dose threshold of 10%. Point dose measurements were also performed using a Pinpoint ionization chamber inserted into the ArcCHECK. Finally, for test 7.5, the MPPG 5.a recommends an external review as a complete end-to-end test for IMRT/VMAT delivery validation. Therefore, to verify the accuracy of the TPS, an end-to-end test was performed using the Australian Clinical Dosimetry Service (ACDS) Audit phantoms [14][15][16] The ACDS audit plans were created, delivered and analyzed following the ACDS Audit speci cations.

Results
The results of the MPPG 5.a validation for the Eclipse photon and electron beam models created for our Varian iX Linac are presented below.

A Basic validation tests
A comparison between some of the measured and TPS calculated beam data for tests 5.   Table 2 shows the results obtained for the differences between measured and calculated point doses for AAA and AXB. The results revealed that most of the measured data agreed well with the TPS calculated data for all the tests and were within the 2% tolerance recommended by the MPPG 5.a.

B Heterogeneity correction validation
A comparison between the CT numbers from the Eclipse generated CT-to-ED curve and those from the scanned electron density CT phantom showed good agreement and the maximum observed difference was 12 HU as shown in table 3. For heterogeneity veri cations, comparisons between measurements and TPS calculated point doses distal to the heterogeneous media are shown in table 4. All calculated point doses are within 3% of the measurements for heterogeneous media for both photon algorithms and all three different energies. Results for electron energies are tabulated in table 5; the differences between the measured and TPS calculated point doses for all electron energies were slightly high, average and maximum differences observed were 5% and 7%, but were however within the 7% tolerance recommended by the MPPG 5.a.

C IMRT/VMAT dose validation
For the IMRT/VMAT dose validation tests, the MPPG 5.a recommends relatively tight evaluation criteria, however there is no widely accepted criteria published for small elds or IMRT/VMAT. All tests were evaluated using 3% / 3mm and 2% / 2mm Gamma criteria. For Test 7.1, the PDD and output were evaluated for very small eld size, and the measured PDD agreed with the TPS calculated PDD within 2% beyond the build-up region for all beams. Fig 6.a and 6.b show an example of gamma analysis between measured and calculated data a PDD and crossplane pro le for a 2x2 cm 2 eld size. For test 7.2, the output was measured using a diode detector and a Pinpoint chamber, and the dose comparisons were within 2%. For test 7.3-7.4, Eclipse TPS calculated dose distributions were compared against the ArcCHECK measurements on two IMRT and ten VMAT plans for prostate, endometrium, pelvis, head and neck, brain, and rectum treatment sites. The results of the Gamma passing rate using criteria of 3%/3mm and 2% /2mm are tabulated in table 6 together with the number of elds/arcs. The average passing rates across all sites using criteria of 3%/3mm and 2% /2mm were 98.8% and 97.7%, respectively, while the average absolute point dose difference was 1%. For Test 7.5, the ACDS audit plans passing rate was over 95% for gamma criteria of 3%/3mm.

Discussion
Veri cation of the accuracy of the dose calculation algorithms is considered an important step in the radiotherapy treatment process. This ensures that the TPS calculated dose meets the prescribed dose and reduces the discrepancies between the TPS estimated dose and the actual dose delivered to the patient [17]. In this work, the AAPM MPPG 5.a validation tests were used to evaluate the accuracy of a new version of the Eclipse TPS algorithms. The MPPG 5.a validation tests were re-arranged in slightly different sections, in three instead of the four sections in the original guideline's recommendations.
In the rst section, tests 5.1 and 5.3 are considered con rmation tests and are straightforward in which they do not require further measurement apart from commissioning data. Results show good agreements between the TPS calculated data and measurements. The commissioning data for all eld sizes and energies are within the recommended speci cations by MPPG 5.a. Test 5.2 veri es the photon and electron beams absolute output under reference calibration condition in the TPS. The results for the photon beam con rm the agreement with measurements and are within the 0.5% tolerance as recommended by MPPG 5.a for photon beams. Results for the electron beam are within 1%. While the MPPG 5.a does not have speci c recommendations for tolerance for the electron beam in this test, the tolerance selected in this work was from the AAPM TG-142. For the static photon and electron validation tests (5.4-5.9 and 8.1-8.2), comparisons between TPS calculated and measured PDDs and pro les were con rmed using the ScanDoseMatch analysis software [5,7,13]. Almost all the tests analyzed yielded gamma index values below 1.0 using a 2%/ 2 mm passing criteria (relative mode) for photons and electrons (Fig 5). For all point dose, the agreement between calculation and measurements were less than or equal to 2% (table 2). Nevertheless, results have also indicated some disagreements where gamma index was greater than 1.0 for some large pro les at the penumbra regions, and these results support previous ndings [11].
In the second section, the heterogeneity correction tests; the accuracy of the CT-to-ED conversion curve is con rmed in tests 6.1 (table 3). For the heterogeneity point dose measurements for photon beam, the Eclipse AAA and AXB algorithms results fall within the tolerance of 3% for the two eld sizes and three energies applied in this test (Table 4). For electron validation test 8.3, the Eclipse eMC algorithm, point dose results were higher, with a maximum difference of 6.8% from the measurements. Though these results were relatively high, they still easily met the 7% recommended tolerance by MPPG 5.a (Table 5).
These results are in agreement with the ndings of the previous results [5,7,11].
In the IMRT/VMAT dose validation section, for tests 7.1 and 7.2, the results for the PDD and pro les scored a passing rate greater than 95% using 2%/2 mm criteria for all energies investigated. The output factor for the small eld was within 2% of the calculated TPS data. Tests 7.3-7.4 were considered as most import tests as these veri ed the beam model accuracy for the Eclipse TPS. All the IMRT and VMAT calculations were only performed using the AXB algorithm as this is what is clinically used in our centre. The results for all IMRT/VMAT cases, the minimum pass rate was 95.6% and 94.3% for 3%/3mm and 2%/2mm criteria, respectively. The average and standard deviation in point dose was 0.9±1%. In our clinic, the minimum pass rate is set at 95% for criteria of 3%/3mm, so based on our local criteria all cases passed. For test 7.5, we followed ARPANSA's ACDS end-to-end veri cation format for external audits and this gave us con dence in our TPS validation and modelling.

Conclusion
The aim of this study was to evaluate the performance of a new version of the Eclipse TPS algorithms, AAA, AXB and eMC using MPPG 5.a. The results obtained demonstrated the accuracy of the Eclipse version 15.6 TPS algorithms, AAA, AXB and eMC, for photon and electron beams. Basic validation, heterogeneity corrections and IMRT/VMAT dose validation for Eclipse's photon and electron algorithms have been demonstrated. The MPPG 5.a validation tests provides the opportunity for the physicist to understand the accuracy and limitations of the dose algorithm commissioned prior to clinical implementation. Comparisons between TPS calculated beam pro les and point doses were performed for three sections. An extensive set of measurements were also performed, which were compared to corresponding TPS calculations. The data collected from the dosimetric performance results obtained during the validations testing is helpful for TPS upgrade or commissioning and for routine TPS QA.        Comparisons between measured and calculated data for tests 5.1 and 5.3, for (a) crossplane pro le of a 6 MV 40 x 40 cm2 eld at 1.5 cm depth, (b) PDD of a 18 MV 40 x 40 cm2 (c) crossline pro le of a 6 MV Flattening lter free (FFF) 10x10 cm2 eld at 1.5 cm depth, and (d) output factor of a 6 MV photon beam.

Figure 5
Gamma analysis between measured and calculated data for tests 5.4 -5.8, for (a) crossplane pro le at 10 cm depth for a 6 MV FFF for test 5.4, (b) crossplane pro le at 10 cm depth for 6 MV for test 5.5, (c) crossline pro le at 25 cm depth for 6 MV for test 5.7, (d) crossplane pro le at 1.5 cm depth for 6 MV FFF photon beam for test 5.8, (e) pro le enhanced dynamic wedge for 18 MV for test 5.9. (f) inline pro le at the depth of d90 (2.8 cm) for a 12 MeV electron delivered through a cut-out shape eld size for test 8.1 (g) a PDD for a 12 MeV electron beam delivered through the standard 10x10 cone applicator at oblique beam incidence for test 8.2.

Figure 6
Gamma analysis between measured and calculated data a PDD and crossplane pro le for a 2x2 cm2 eld size.