AI] 21 Sep 2023 arXiv:2309.11805v1 [c: JobRecoGPT: Explainable job recommendations using LLMs Preetam Ghosh, Vaishali Sadaphal Intelligent Service Networks, Tata Research, Tata Consultancy Services Abstract—In today's rapidly evolving job market, finding ‘the right opportunity can be a daunting challenge, With advancements in the field of AI, computers can now recommend suitable jobs to candidates. However, the task of recommending jobs is not same as recommending movies (o viewers. Apart from mustchave criteria, like skills and experience, there are ‘many subtle aspects to a job which can decide if it is a good {it or not for a given candidate. Traditional approaches can ‘capture the quanlifiable aspects of jobs and candidates, but a substantial portion of the data that is present in unstructured form in the job descriptions and resumes is lost in the process cof conversion to structured format. As of late, Large Language ‘Models (LLMs) have taken over the AT field by storm with ‘extraordinary performance in fields where textased data is available. Inspired by the superior performance of LLMs, we leverage their capability to understand natural language for ‘capturing the information that was previously getting lost during ‘the conversion of unstructured data to structured form. To this ‘end, we compare performance of four different approaches for job recommendations namely, (i) Content based deterministic, (i) LLM guided, (ii) LLM unguided, and (iv) Hybrid. In this study, we present advantages and limitations of each method and ‘evaluate their performance in terms of time requirements. I, IntRopucTiON Identifying job opportunities for talent is important to ‘enable organisations to attract, develop, and retain the talent. It is a time-consuming process when done manually and results in limited reach, inconsistent criteria with human bias ‘With the emergence of freelance platforms and integration of technology, significance of data-driven job recommendations hhas grown, ‘Traditional state-of-the-art techniques recommend job opportunities to the talent based on similarity between job requirements and talent attributes, However, the nature of data in this domain is inherently in unstructured natural language format viz. resume and job descriptions. To use traditional approaches, one is required to extract information and bring it to a structured formal. In this study, we investigate the application of large language models (LLMs) [16] due to their ability to process and comprehend language, as well as their extensive knowledge gained {rom training on Internet text. Providing recommendations falls in the category of Reasoning intelligence, specifically Prescriptive intelligence. Generative AI and language models are observed to be performing well in Recognition and Operative intelligence. In this study, we investigate role of language models in Reasoning intelligence. ‘The traditional methods are required to define a structure that consist of requirements of jobs and corresponding attributes of talent such as role, skills, educational background, ‘and experience, among others. The jobs with closest match ‘with the attributes of talent are recommended, These methods face several challenges. + The task of transforming data from CVs and JDs into a structured format is error-prone and can result in loss of information [1] + The qualitative aspects of talent such as achievements, strengths, and aspirations are ambiguous and presented in natural language, hence not extracted + The data driven tools rely on quantitative metrics, such as skills and experience, while overlooking qualitative aspects like soft skills oF potential for growth. + The JDs may be incomplete, the qualitative aspects of the job and organization may be biased or even may not be ‘mentioned, In this work, we leverage the capability of LLM’s to understand natural language for capturing the information that was previously getting lost during the conversion of ‘unstructured data to structured form. We present, + One content based deterministic approach: based on traditional techniques. This is used as a baseline (0 compare performance of all approaches + Two LLM based approaches: guided and unguided ‘+ A hybrid approach: a combination of traditional and LLM based approach + Evaluation: Comparison of quality of recommendations produced by each method and the efficiency ‘Though the approaches proposed in this work are generic, ‘we consider the domain of Information Technology to evaluate the effectiveness of these techniques. We conduct experiments using two datasets + Synthetic dara: In this experiment, we generate synthetic ddata that simulates the characteristics of the IT domain, allowing us (o assess the performance of all methods in controlled environment. ‘+ Real world data: Yn this experiment, we use real IDs from the IT field. This enables us to evaluate the practical applicability and performance of the methods using authentic data, ‘Through these experiments, we aim to gain insights into the effectiveness and limitations of the all the techniques for providing ranked job recommendations for a talent in the IT domain,Fig. 1: Unstructured Resume CV3. II, CONTENT BASED DETERMINISTIC APPROACH, Deterministic approach is on the lines of content based techniques of recommending jobs to talents. This simple approach is used as a baseline to compare performance of all approaches. In this approach, a job is recommended by ‘matching its requirements with the aitributes of talent Refer Figure 4, The algorithm accepts inputs as unstructured resume (CV) and job descriptions (JDs) and configurations that, include objective direction, indicating if higher, lower or closer values of the job attributes with respect to the talent attribute is better and the number of recommendations required. The ‘output is the recommended JDs with a score 1) Unstructured to structured conversion: Figure | and 11 show a CV and a JD, respectively. Figure 2 shows the attribute ‘model for a talent and a job. The talent attributes capture talents professional information and preferences. Whereas, the job attributes capture corresponding requirements. ‘The talent and job model is populated by extracting information from the CV and JD. The structured model of talent and job corresponding to the CV and JD in Figure | ‘and 11 is shown in Figure 3, and 10 respectively. We use LLMs to convert unstructured data to structured form. The prompts used for this conversion are depicted in Figure 12, 2) Deterministic algorithm: The deterministic algorithm, as the name suggests, generates predictable, hence reproducible results every time, based on the ctiteria provided to it. It provides a baseline score to compare recommendations from ese ate Suter tas nsrora pee Fig. 2: Attributes of talent and job, pal sping ot ors Ut ong Frameworks (it, “Unt, “Mock, act 0, a fe b “nr ana L anieaton" L *euconal dere" L ‘ole paternc L "expres in tle Fig, : Structured resume CV3, other approaches. Though there are complex recommendation approaches, we keep it simple while capturing the required attributes that determine if a job is suitable for a given talent. At the core of this algorithm, ate basic comparisons between various attributes extracted from the CV and ID. ‘+ “Closer” match: A rating of 1 is awarded with absolute deviation equal to zero and the rating decreases as absolute deviation increases. This approach penalizes ‘both under-qualified candidates, It also penalizes over-qualified candidates as they may be better suited for other roles. + “Exact” match: A score of 1 is awarded if a value or a set are an exact match. This is important for some attributes where we a specific value is required, say mandatory certification, experiment, we configure all the attributes set to ” match, the only exception is the “certifications” for which we set it to “exact”. Consider the following example. 1) Skill proficiency: Consider that three skills out of four ‘are common, with one having same proficiency and the other two with a deviation of 2 from required. This leads to-a score: (1 +1/2+1/2)/4 = 0.5. The first value is 1 as the proficiency exactly matches with the requirement and the next two values have 1/2 as the values deviate by 2. 2) Time zone: It is an an absolute difference in hours and is calculated ranging between 0 to 26 which is then normalized between I and 0,= = as ——Ee Fig. 4: Deterministic algorithm. 3) Certification: It is a set match of the required certifications. The score is 1 for a complete set match, else 0, 4) Education; It is denoted as a number from 1 to 5 with higher value for higher education, Its score is the reciprocal of the absolute difference with respect to requited value 5) Experience: The score is calculated as a ratio of talent value and required value in the required role. For ‘example, for the matching role the actual value is 6 and required is 10, resulting in score of (6/10) = 0.6. 6) Role: Itis the reciprocal ofthe preference of the required role by the talent. For example, the candidate has 2nd preference for the role, resulting in a score of 0.5. 7) Finally, an average value of all scores is (0.5 + 141+ 1406 + 0.5)/6 = 46/6 = 0.7 II]. LANGUAGE MODEL BASED APPROACH ‘Though large language models (LLMs) are predictors of, next words and lack true comprehension or knowledge, they have a convincing ability to generate coherent responses and recall information. This makes it seem like they possess knowledge of many domains. We propose to leverage this ability of LLMs to understand and correlate different required skills and roles based on their similarity. For example, Someone experienced in design engineering and software development in the domain of IT may inherently be suited for 1 Full stack developer role, even if not explicitly mentioned. This provides a significant advantage over methods that are constrained by specific value and text based similarities. We provide unstructured CV and IDs to a language model to generate job recommendations. Refer Figure 5 and 6 for the two LLM based approaches. We use GPT4 [2] model of ‘OpenAL for this purpose. A. LLM Guided Algorithm Here, we provide a pre-defined structure to generate recommendations. This allows control over the model's response, such as the criteria used for matching, the number of recommendations, and returning recommendations in a structured format, Further, we exploit LLMs ability to explain their actions in natural language to provide reasoning of why ‘a certain recommendation is good or bad, Figure 5 shows the overall flow and prompts provided as input to LLM. A configuration is provided to “guide” the LLM. according in the desired objective direction, “criteria”. This ESE mtr Fig. 6: Language model based unguided algorithm. information together with appropriate prompts is provided as fan input to the LLM. The output from the LLM is consistent due to the guidelines from the prompts and consists of recommended Job ID along with explanations such as benefits, drawbacks, and qualitative aspects. B. LIM Unguided Algorithm To leverage LLMs ability to comprehend natural language, wwe perform an experiment where no clear criteria is provided for recommending a job to a talent viz. an “unguided” approach. This way, the LLM model is expected to recommend jobs that are ‘good’ according to its own comprehension. We direct the model in the prompt to add an explanations for the recommendation. With this approach, the model provides recommendations with logical explanations. Figure 6 shows the overall flow and prompts used in this approach. This time no “criteria” is provided to guide the rodel. The output oblained from this approach does not follow any specific structure. Nevertheless, it provides a structure t0 its response in multiple paragraphs with the Job ID at top and the explanations following below as a bulleted list.. Handling large data LLMs have limitation on number of tokens that can be provided as an input. It is 8192 for the OpenAL GPTS. When working with a large number of unstructured JDs, it is inevitable that we wall hit this token limit. In that cease, we propose to: () Split the JD set to smaller subsets (i) Get recommendations separately from each subset (ii) Merge the top recommendations from each subset, Note that this merging only top recommendations fom each set will overlook superior but lower ranked recommendations from other sets IV. Hyarp approach ‘The key idea here is to complement weakness of one technique by the strength of the other. + Qualitarive aspects: Traditional technique is effective Jn matching well-structured quantitative atributes such as skill proficiency, role, experience, among. others, However, they lack in capturing qualitative aspects of talent and job. Whereas, LLM can comprehend language and take into consideration qualitative attributes and soft skills that are important for a job but that have been missed inthe structured model. Further, they can provide justifications of recommendations in natural language format. + Unbiased view of job and organization: LLM can provide an unbiased view of a job and the organization based on its knowledge gained from training on Internet data. This can guide the user to take an unbiased informed decision, + Scalability and cost: LLM based technique faces a drawbacks tha it is costly and has scalability challenge due to limitations on number of tokens. Whereas, the traditional technique is computationally efficient and fas, hence has the capability to process large amount of data We propose a hybrid approach that combines traditional method with use of language models to generate richer job recommendations. The key-idea is (i) to use traditional method to trim down large number of job opportunities to a smaller relevant set and (i) prioritize these further on the basis of qualitative aspects, with well justiied reasons, benefits, and drawbacks in natural language. Figure 7 outlines the hybrid approach. Deterministic algorithm is used in the first stage and the unguided LLM method is used in the second stage ‘The recommendations are further enriched by an unbiased view of the role and organization to overcome the issue of IDs of organizations highlighting only the perks of working for them, without mentioning drawbacks. An unbiased view ‘makes it easier for the candidate to make an informed decision. ‘We prompt the LLM to rate the organizations and the particular role in that organization on a scale of 1 to 10. The prompt containing the aspects of the organization and roles that are considered for the rating are shown in Figure 8 We derive recommendations on two sets of dala viz. (i) controlled experiments using synthetically generated data (i) experiments using real world JDs. We present the results in ‘upcoming sections. Fig. 8: Prompls used to generate job and organization ratings \V. EXPERIMENTS AND RESULTS: SYNTHETIC DATA, A, Data generation We model the domain of Information Technology by populating relevant values of attributes, The list of values and their ranges are shown in Figure 9. We sample from these values to populate structured models of talent and job (Figure 3, 10) For the purpose of performing controlled experiments, we ‘generate ten JDs for every CV. The jobs are generated in a way such that, one JD is almost an exact match, three IDs deviate a little, the next three have larger deviation, the last three fare quite different. The uaditional deterministic algorithm is expected to recommend and rank the IDs in above order. We generated realistic unstructured CV and JDs from the structured talent and JD models using GPT4. These provided Sr epee Fig. 9: Set from which attibute values ate sampled"at Fig, 11: Unstructured synthetic IDs: JD7 and JD3, as input to LLMs. The prompts used for this purpose is shown in Figure 12. Figure 11 shows unstructured JDs corresponding to structured JDs in Figure 10, B. Results We present details of recommendations generated by each algorithm for one candidate (CV3). Refer Figure 13 for recommendations by deterministic algorithm, JD7 is a perfect match for CV3, with all attributes matching and just slight variation in the skill profciencies. 1D9 is the next best match, requiting better skill proficiency, and more certifications ‘The language model pays more attention to the role ‘match, experience and education. Refer Figure 14. for recommendations by “guided” approach. It has recommended JD3 which isn’t a good match by deterministic algorithm, with only the role requirement and timezone conditions being satisfied. However, it as weighted the qualitative attributes that are not captured in the structured format viz. collaborative and inclusive environment, opportunities for skill evelopment and career advancement. One aspect it seems to ris out across all its recommendations is the mismatch in representation of proficiency level. It is unable to understand the terminology of beginner, intermediate, advanced level and See gee rere 82H, MEN Ma etna: Erase on cid SM eda Sena ‘Ste wth cee geen ena ermine en te tn Fig. 12: Synthetic data generation prompts for creating unstructured CV (top) and unstructured JD (bottom) 7610 Seore 107, 092 109) 077 106 O74 104. 0.69 103, 0.45 Fig. 13: Deterministic algorithm recommendations for synthetic CV3 compare it with numeric proficiency levels between 0 and 5. This behaviour is expected as domain dependent numeric values do not make sense unless the range of the values or its ‘mapping with textual classifications are mentioned in advance. thas recommended all the JDs that have preferred role by the candidate with CV3 viz, Full Stack Developer and Technical Lead, Refer Figure 15 for recommendations by “unguided approach. In this case, 1D4, JD6, JD7 have been recommended and the recommendations are well justified. Again, it has given more preference to role, and experience. The qualitative aspects of organizations is correctly captured and presented in greater detail. Being completely unstructured, it clearly explains the perks of the recommended jobs and also ‘mentioned the reasons why the others were not a good match. Figure 15 presents recommendations generated by the hybrid algorithm, The deterministic algorithm recommends 5 IDs, The LLM unguided method then picks out the best 3 from among those. In addition to this, the algorithm also provides ratings of qualitative aspects of organizations and job traits as shown in Figure 17. Google and its Full Stack Developer position is rated highest 9 on 10 while SAP has organization rating of 8,7 and its Full Stack Developer position has a rating of 8.6. Finally, TCS has organization rating of 8.5 but the position of Technical lead is rated 8.7. This helps the candidate have an unbiased view and take an informed decisionFig. 14: LLM guided recommendations for synthetic CV. mine sr rent Gong Grameen sen a0 company cure: 910 Aegan di 0 Trepareneconmanaton 0, ncaa 0 eng esfie ralersing 0 pat on ay: 920 ek th ating age etnies 0 emate fea tierce 90 Inorton ad rent 0 (reer saarconem 90 {ease fevtopmetOpprntie 810 Impact jes 0 Team Oyu 90, Cluberton ad Fama 9/10 ot eerie roth 910 incu ecg penton on eres: 8/0 ‘shear nda 0. ‘ett tee 0 Fig. 17: Organization and job ratings in hybrid algorithm recommendations for synthetic CV3. Emptatzon iy 9 tot tinge edinee 9 Impact jee» Tenors? Fig. 15: LLM unguided recommendations for synthetic CV3, Fig. 16: Hybrid recommendations for synthetic CV3, eae Scans atest en nce cas yoy et wg ete red LEE sec sot Fig. 18: Prompt for unstructured CV generation from a real ‘unstructured JD, VI. REAL WORLD EXPERIMENTS A, Data description We used real JDs from Kaggle [3] and generated CVs from these, The unstructured CVs are generated using LLM. The prompt used for this is presented in Figure 18. The JDs have 4 Tot of variation, which helps capture real world situations where IDs can be completely different, ‘We used five CVs to generate recommendations, For three of the CVs, we did not include the source JDs from which they are generated, whereas for two CVs, we included the source JDs. The thought behind this was to have some CVS which would not have a one to one match with any of the available JDs and the recommenders would be forced to pick some next best option, This would help make this experiment more relevant and better suited to capture real world scenatio, B. Results We present details of recommendations generated by each algorithm for one CV (CV2) shown in Figure 19 and 20. Figure 21 show two unstructured JDs and Figure 22 shows comesponding structured JDs. Figure 23, 24, 25, and 26 show the recommendations by deterministic, LLM-guided, LLM-unguided, and Hybrid approach respectively.Fig. 19: Unstructured real CV2. ‘The deterministic and LLM approaches recommend JD9 from which CV2 was generated. It is best with respect to skill match and role of “DevOps Engineer”. In was observed that none of the JDs had any specific location requirement. AS a result, LLM took liberty to show it as either a drawback for a benefit. In the LLM-guided approach, the method brings it out as a drawback. However, the LLM-unguided approach and Hybrid approach bring it out as a benefit, LLM-guided approach points out that the job opportunity is not at the preferred location "San Francisco”. However, LLM-unguided ‘approach and Hybrid approach mention that there a match in preferred location, Hybrid algorithm brings out the positives ‘of JD9 with respect to the skill match and how particular skills such as Hadoop, Urban code, Tomeat will help in “Infrastructure as Code in a DevOps oriented organization” Rest of the JDs recommended are not as good a match ‘compared to JD9. However, JD6 has been recommended on the basis of similarity of skills of DevOps Engineer and Java Developer or Software Developer position Figure 27 shows the job ratings produced by LLM in Hybrid algorithm. Clearly, the position of "DevOps Engineer” in D9 hhas a higher rating of 7.6 compared to a “Java Developer” position in JD6. VII. BALLPARK ESTIMATES OF QUALITY OF ALGORITHM, Assigning a quality score to a recommendation without hhuman feedback is hard, We assign reference quality scores ‘manually to cach recommendation. The manual reference scores are assigned based on how a human would have rated the jobs for a given CV, ie, by considering all “must-have” conditions first, like skills, education, experience and then focusing on the “good-to-have” like location, certification and other associated benefits provided in the job package. However, since these scores are manually assigned there can Fig. 20: Structured real CV2 be oversights along with aspects of jobs that can be interpreted differently depending on the individual rating the jobs. So, rather than treating the scores as ground truth, we use them as guidelines and assume thal some deviation from them is acceptable. ‘The quality scores of algorithms (Figure 28 and Figure 28) are computed as an average of absolute deviation of algorithm, scores from the manual reference scores, In case of LLM ‘unguided algorithm that just provide rank of recommendation the algorithm score is assigned a value same as the manual reference score for the same rank, A, Synthetic data Figure 28 shows our findings. The results from the four ‘methods exhibit striking similarities, Among them, the LLM ‘unguided approach stands out for its superior performance, closely resembling human-friendly recommendations, and having a accuracy of 95.66%. We observed that LLMs excel in utilizing domain knowledge and capturing unstructured clements, showcasing their greatest strengths. Conversely, it is evident that constraining the LIM in guided approach yields the poorest results among the four methods, with ‘around 93.66% accuracy. Deterministic method is better suited for large scale, rendering the use of LLMs inefficient and “unjustifiable in terms of time and cost. Consequently, Hybrid approach, leverages the unguided LLM method in conjunction ‘with deterministic methods to combine the best aspects of both, approaches, generating results almost identical to the best, with an accuracy of 95% which is just shy of 0.66% from the best. B. Real world data (Our findings for recommendations for real world data are depicted in Figure 29. The quality score on real data is somewhat lower compared to the synthetic data, Similar to the synthetic data, LLM unguided once again outperforms all other methods with an impressive accuracy of 89.66%,jobId Score 109 052 106. 031 102, (026 102, 025 104. 025 ig. 28: Deterministic algorithm recommendations for CV2. ig. 21: Unstructured real JD6 and JD9. Fig, 22: Structured real JD9 and ID6. ‘The hybrid method secures the second-best position with ‘an overall accuracy of 87%, following the same tend as before. LLM guided, on the other hand, falls in the lower half with an accuracy of 84.66%, marginally oulperforming the deterministic method which exhibited the poorest performance this time. The subpar results of the deterministic approach highlight the loss of information during the conversion (© a Fig, 24: LLM guided algorithm recommendations for real CV2, structured data format ‘An additional interesting observation can be made from these results. There is a noticeable decline in performance for ‘one of the CVs, CV4, primarily due to the absence of an exact job match, Recommendations for this talent had to rely solely ‘on other relevant factors and general knowledge to suggest a suitable job. Interestingly, all the methods performed equally poorly on this task. In conclusion, itis evident that LLM unguided exhibits the best performance. However, for practical purposes and the ability to handle larger data volumes, the Hybrid algorithm [Reston par inated a inane opened SSE cm gu ppt no Sierra seston pyar on oer Fig. 25: LLM unguided algorithm recommendations for real CV2.ig. 26: Hybrid approach recommendations for real CV2 Fig. 27: Job ratings for positions recommended for real CV2. Fig. 28: Quality estimate of recommendations on synthetic data roves to be a more suitable choice. VIII. PERFORMANCE ANALYSIS We evaluate runtime performance of all the approaches. We observed a strong correlation between the number of tokens used in the LLMs, both input and output, and the time it takes to generate recommendations ‘+ Deterministic algorithm: In this case, the time taken by the algorithm itself is negligible when compared to the time requited to convert unstructured to structured data (On an average around 20 seconds are required to convert a given CV or a JD to structured format, However, in real. Fig, 29: Quality of language based recommendations on real data IDs may not be required to be converted to structured format every single time the recommendation system runs, + LLM guided and LLM unguided: Both of these methods exhibit similar time requirements, although LLM. unguided takes slightly longer due to its tendency to employ verbose reasoning, resulting in a higher token count during the process, Generating recommendations for a given CV typically takes an average of 25 to 35 seconds using either method, ‘+ Hybrid: The hybrid method combines both deterministic and LLM approaches, executing the process sequentially, which results in the cumulative time required by both methods. To run the deterministic part and select good recommendations, it takes approximately 20. seconds. Subsequently, the LLM part is invoked, which takes an additional 25 seconds, resulting in a total time of approximately 45 seconds for a single recommendation, While this duration may appear large, it should be noted that even with hundreds of jobs to recommend from, the time will remain zelatively constant IX, LiTeRATURE REVIEW Recommendation systems find extensive use in a variety of domains such as E-commerce, streaming platforms, due to their ability to enhance user experiences and drive engagement. They are generated using various techniques, such as collaborative filtering, content-based filtering, oF hybrid approaches. In the past, these techniques have been used for the purpose of recommending job opportunities to talent as well Content based recommendations ake into account the semantic similarity of attributes that are present in the CV and IDs. ‘There are a number of approaches on how these similarities are calculated. In [11] Bag of Words method is used, whereas in [4] uses the Latent Dirichlet allocation algorithm. We also find relatively new methods such as ‘word2vee being used for job recommendations [9], {15} Tn Collaborative filtering the recommendations are generated by considering the previous selections a candidate hhas made and recommending similar jobs in the future. Memory based collaborative filtering is predominantly used for job recommendations where candidates are grouped based on the similar interactions. Jobs are recommended to candidates by the preference of other candidates belonging to the same group. [12] and [13] apply this method. ‘This is by far the most common approach for job recommendation. Contributions of Model-based methods on shallow embeddings can be found in [6]. In recent times Deep neural network based methods have gained a lot of traction with [I0]using them for the purpose of job recommendation, There are also ensemble hybrid methods [7] and weighted hybrid which assigns some weightage ‘o the different contributing methods [14]. Apart from these some notable works in this field include [S]which talks about thedownside of too many recommendations and the impact of ‘wrong recommendations on a user, Finally, knowledge based recommendation are based on using inherent knowledge about the job in form of an ontology and it ean recommend the jobs to candidates once they too are ‘categorized following the same ontology. This is discussed in more details in [8]. ‘The LLM based approach circumvents issues related to information extraction and modelling. The Hybrid approach ‘generates predictable recommendations that ensure quality expected by a user, further enriching them by the use of LLMs by capturing attributes that were not captured in the model, by removing bias, and providing justifications X. ConcLusion Unstructured data is prevalent in job recommendations, as CVs and JDs (JDs) are often in unstructured formats Converting unstructured data into structured formats has been the traditional approach, but achieving accurate conversion remains challenging. Large language models (LLMs) have emerged as valuable tools in this field. However, LLMs can be slow, expensive, and lack full controllability, resulting in indeterminism and occasional inaccuracies in their results, To improve job recommendations, we propose a hybrid approach that combines quantitative attribute filtering followed with LLM analysis. By initially filtering out irrelevant jobs based ‘on quantitative attributes, we can reduce the volume of data before utilizing LLMs to consider the qualitative aspects of the remaining screened jobs, Our experiments demonstrated that relying solely on LLMs may result in job recommendations that lack the required attributes for candidates. Deterministic methods can also overlook attributes that have not been explicitly modelled and miss important qualitative aspects In contrast, the hybrid method performs well in terms of clliciency, cost-effectiveness, and capturing both quantitative ‘and qualitative aspects, Additionally, IDs often exhibit bias ‘and highlight only positive qualities. Providing an unbiased perspective of the organization and job position based on the common knowledge of LLMs can empower candidates to make informed decisions. In conclusion, the LLM-based approach adds value to talent acquisition. It is recommended to be used in conjunction with traditional algorithms to handle scalability and address specific requirements. XI. Discussion ‘The problem of recommending jobs faces multitude of challenges. Accurate matching of the skills and requirements of the talent with the job opportunities available is hard. Many ID are vague and poorly defined, making it difficult to identify the tight match. Limited information available about the talent, such as their skills, experience, preferences and complex career goals makes it challenging to recommend personalized job opportunities that align with their qualifications and aspirations. Keeping up with the evolving job market and ‘ensuting that the recommended opportunities remain relevant is a significant challenge. Obtaining feedback from talent about the effectiveness of recommendations is essential for continuous improvement. However, gathering feedback requires establishing reliable mechanisms to measure the success and quality of recommendations which is inherently ambiguous. Further, a real world job marketplace, requires to address complex aspects such as when to recommend, how many recommendations are to be made, how the recommendations influence the user choice, implications of ‘a wrong recommendation and many more. By fine-tuning prompts, incorporating few-shot learning, training the model on domain-specific data, and leveraging user feedback and reinforcement learning, it is possible to generate more relevant and personalized recommendations in the future, An important direction of work is to define quality of recommendations unambiguously, possibly using benchmarks. REFERENCES In} upetdocparserconvblogiataexiraction-toate [a] upstopenascomigpt-a IB] pein kagalecomidalastksiizzegmiobs-andjob- description [4] Stivam Baasal, Aman Srivastava, and Anja Arora Topic modeling oven content based jobs recommendation engioe for recruitment Indust. Procedia computer scence, 122:865-872, 2017 Fedor Boriysk, Liang Zhang, and Keishnaram Kenthapadh, Lier ‘A sstem for job application redstibution towards eficient eatee imatkeiplace. In Proceedings of the 25rd ACM SIGKDD Interntional Conference on Knowledge Discovery and Data Mining, pages 1397-1408, 2017 [61 Wein Chea, Xingming Zhang, Haoxiang Wang, and Hongie Xu Hybrid deep collaborative filtering for Job recommendation tn 2017 2rd IEEE Intemational Conference on Compustinal Intelligence and ‘Applications (ICCIA}, pages 219-280. IEEE, 2017 I71 Toon De Pessemier, Kris Vanhecke, and Luc Marens, A scalable Aighperfomamnce algorithm for bybeid job recommendations. — In Proceedings of the Recommender Systems Chalenge, pages 1-4 2016 [8] Mausieio Nons Free and Leandro Nanes de Casco, e-reruitnent recommender systems: a sytemuti review: Knowledge and Information Systems, 63:1-20, 2021 ‘Akshay Gognani and Hemant Misra. Implicit sills extraction wsing ‘document embedding abd is use in jo recommendation. In Proceeding? ‘of the AAAL conference on artificial intligence, volume 34, pages T2u6=13093, 2020, 10} Tons ang, Songyun Ye, Wei Wang, Fingrsn Xv, and Xisosbeng Loo. Learning eftecive representations for etionjob ft by feature fusion. 1a Proceedings ofthe 23th ACM Intemational Conference om Information 4 Knowledge Management, pages 2549-2556, 2020, Rémy Kessler, Nicolat Bechet, Mathieu Roche, Suan-Manuel ‘TteeMoreno, and Mare El-Béze. A hybrid approach to managing job ollerr and candidates Information procerting & manager, ‘(9 :1124-1135, 2012 12] Emansel Lacie, Matkas Reste-Haus, Dominik Kowald, Manoj [Reddy Dareddy,Junghoo Cho, abd Elisabeth Lex. Using autoencoders for seston-hased ob recommendations, User Modeling end UsersAdoped Interaction, 30:6.7-558, 2020, 13] Yuin Lee, Yeon-Chang’ Lee, Jiwon Hong, abd Sang-Wook Kim [Exploiting job anstion patterns for effec job recommendation. In 2017 IEEE Interatonal Conference om Syste, Man, and Cybernetics (SMC), pagee 2414-2419, IEEE, 2017 Vasily Tekin, Aodeey Oxapet, Milhail Kamensikey, Dmiey Khodako, and Vasily Rublow. Combination of content-based ser profiling and loeal elle embeddings fr job recommendation. 2017 Jorge Carlos Valverde-Rebara, Ricardo Pama Pasl Bustos, and [Natalia € Silva, Job recommendation based on jobseeker ake: An ‘empirical stody in TexaStors@ CIR, pages 47-51, 2018, 16} Wayne Xin Zhao, Kun Zhou, Jusyi Li, Tanyi Tang, Xiao Wang YYopeng Mov, Yinggian Min, Beichen Zhang, Tune Zhang, Ziean Dong. eal A suvey of lage language models. arXiv preprint (akin 2903 18223, 2023, 5 14) 15)